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SUMMARY 


Many  Havy  tasks  depend  on  the  skilled  performance  of  highly-trained 
individuals.  Training  for  such  tasks  often  requires  closely  supervised, 
intensive,  and  highly  technical  Instruction  from  a  knowledgeable  instructor. 

In  many  cases,  skilled  Navy  Instructors  are  In  short  supply— and  the  quality 
of  Instruction  can  vary  from  individual  to  individual. 

Recent  advances  in  computer  science  and  psychology  have  created  the 
opportunity  to  develop  "intelligent"  computer-based  instructional  systems. 

Such  systems  can  not  only  identify  trainee  errors  during  instructional 
sessions  but  also  understand  why  the  trainee  has  erred  based  on  its  knowledge 
of  the  task  and  a  model  of  the  trainee's  cognitive  processes.  Such  diagnoses 
are  critical  to  correcting  a  trainee's  misunderstanding  or  specific  skill 
deficiency  about  the  task  he  Is  trying  to  learn. 

The  development  of  such  computer-based  instructional  systems  requires  the 
accumulation  and  codification  of  extensive  knowledge  about  the  target  task  and 
trainees'  behavior  when  learning  the  task.  Such  knowledge— essentially  that 
which  an  expert  instructor  brings  to  bear  during  training— includes  the  skills 
and  procedures  required  co  perform  the  task  correctly,  the  types  of  errors 
trainees  can  exhibit  during  learning,  techniques  for  diagnosing  trainee 
performance  deficits,  and  instructional  interventions  to  correct  skill 
deficits  and  train  appropriate  behavior. 

Development  efforts  aimed  at  constructing  such  knowledge-intensive 
"expert"  systems  often  adopt  the  "knowledge  engineering"  paradigm.  Knowledge 
engineering  is  an  iterative  process  with  six  principal  phases.  These  phases 
include  definition  of  the  system's  capabilities,  extraction  of  domain 
knowledge,  formalization  of  the  knowledge,  design  of  the  expert  system, 
implementation  of  the  system,  and  test  of  system  performance.  The  process  is 
distinguished  by  the  necessity  of  repeated  and  extensive  interactions  between 
a  domain  expert  and  a  highly  trained  system  developer  who  elicits  and  encodes 
the  expert's  knowledge  about  task  performance  and  training  procedures. 

The  shortage  of  experienced  knowledge  engineers  and  the  difficulty  of  co¬ 
locating  knowledge  engineers  and  domain  experts  for  extended  periods  of  time 
limit  the  opportunities  for  the  development  of  new  expert  training  systems. 
Therefore,  a  need  exists  to  develop  automated  tools  for  knowledge  acquisition 
and  formalization  that  could  interact  directly  with  domain  experts  and  reduce 
the  involvement  of  knowledge  engineers.  Such  tools  could  facilitate  the  more 
rapid  and  widespread  development  of  advanced  training  systems  with  automated 
Instructional  features.  These  systems,  in  turn,  would  contribute  to  Navy 
training  effectiveness  by  Increasing  the  availability  of  high-quality 
instruction  through  the  use  of  computer-based  training  aids. 

The  generality  of  current  tools  for  automated  knowledge  acquisition  Is 
limited  both  by  the  present  state  of  the  art  in  knowledge  engineering  and  by 
dependencies  among  stages  In  the  process  that  make  It  difficult  to  isolate 
individual  functions.  The  process  of  acquiring  domain-specific  expert 
knowledge  Is  dependent  on  the  formalisms  used  to  represent  that  knowledge, 
which.  In  turn,  depend  on  how  the  knowledge  will  be  used.  The  technology  for 
engineering  knowledge-based  systems  has  not  yet  progressed  to  a  stage  In  which 
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relationships  among  domain  characteristics,  system  objectives,  and  knowledge 
representation  requirements  are  well  understood.  This  lack  of  understanding 
limits  the  generality  of  system  architectures  and  of  any  potential  aids  for 
the  system  building  process.  To  a  large  extent,  the  power  of  such  aids 
already  developed  has  been  Inversely  related  to  their  generality. 

The  design  of  automated  aids  is  also  complicated  by  the  opportunistic, 
unpredictable  nature  of  the  system  development  process.  The  opportunistic 
nature  of  the  process  derives  from  the  multiple  dependencies  among  the  various 
stages  in  system  development.  The  system  builder  has  a  number  of 
interdependent  objectives  in  addition  to  domain  knowledge  acquisition,  and  he 
must  pursue  them  iteratively  and  Incrementally-- that  is,  in  parallel— since 
there  is  inadequate  design  knowledge  for  achieving  them  sequentially.  It  is 
difficult  to  design  an  automated  aid  to  be  used  under  such  variable, 
unpredictable  circumstances.  These  characteristics  of  the  knowledge 
engineering  process  limit  the  scope  and  generality  of  a  knowledge  acquisition 
system  that  can  be  produced  given  the  current  state  of  the  art. 

Within  these  constraints,  project  research  considered  a  number  of 
alternative  concepts  for  aids  to  assist  the  development  of  knowledge-based 
instructional  systems.  These  concepts  were  evaluated  against  a  number  of 
criteria  that  considered  their  feasibility,  utility,  and  appropriateness  for 
use  in  military  applications.  The  most  promising  concept  entailed  the 
development  of  systems  for  the  acquisition  of  knowledge  in  a  variety  of 
domains  with  a  single  conceptual  class.  The  notion  of  a  class  is  somewhat 
intuitive,  but  it  may  be  defined  more  technically  as  a  set  of  domains  that 
share  a  significant  number  of  concept  abstractions  among  their  bodies  of 
knowledge  and  that  could  be  taught  by  a  single  training  system  with  a  fixed 
set  of  instructional  features.  Such  classes  within  the  Navy  include  sonar  and 
radar  system  operations,  system  maintenance,  and  platform-level  combat 
tactics. 

The  development  of  an  automated  knowledge  acquisition  aid  generic  to  a 
class  of  domains  requires  the  prior  development  of  an  adequate  training  system 
for  one  domain  in  the  class.  Subsequently,  the  automated  aid  is  developed  and 
used  to  implement  the  "same"  training  system  for  other  domains  in  the  class. 
The  automated  aid  depends  on  the  particular  Implementation  of  the  training 
system  and  on  knowledge  it  embodies,  abstracted  from  the  first  development 
effort,  about  how  to  elicit  and  organize  domain  knowledge  for  the  domain 
class.  This  approach  avoids  the  problem  of  Interfacing  an  aid  to  an  iterative 
and  variable  set  of  activities  by  completing  all  the  activities,  except  the 
acquisition  and  encoding  of  specific  domain  knowledge,  in  the  course  of 
building  the  system  for  the  first  domain.  The  finished  results  of  those 
activities  are  transported  intact  to  the  systems  for  the  other  domains.  The 
approach  follows  from  and  builds  upon  the  techniques  used  in  the  "skeletal 
systems"  developed  to  assist  Implementation  of  expert  consultation  systems  for 
specific  classes  of  problems. 

This  report  describes  three  alternatives  for  Implementation  of  a  class- 
generic  knowledge  acquisition  system  for  use  In  developing  advanced  training 
systems.  The  first  would  implement  an  architecture  and  knowledge  acquisition 
aid  for  eliciting  from  an  expert  the  knowledge  needed  to  perform  automated 
performance  diagnosis  of  a  trainee  during  learning.  The  automated  aid  would 
acquire  an  instructor's  knowledge  of  potential  errors  in  task  performance  and 
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other  performance  deviations  needed  to  support  automated  performance  diagnosis 
during  training  sessions. 

The  second  alternative  knowledge  acquisition  aid  would  capitalize  on 
ongoing  efforts  to  develop  “intelligent"  automated  opponents  in  tactics 
training  systems.  In  this  case,  the  aid  elicits  elaborations  of  alternative 
(possibly  sub-optimal)  opponent  tactics  and  knowledge  of  when  during  training 
to  invoke  these  alternative  behaviors  to  achieve  pedagogical  objectives. 

The  third  alternative  extends  a  prior  Navy  research  and  development 
effort  that  implemented  a  prototype  generic  instructional  system.  That  system 
uses  instructional  "games"  to  assist  trainees  in  the  memorization  of  domain 
facts  and  relations.  It  has  already  been  applied  to  several  different 
domains.  However,  building  new  domain  knowledge  bases  for  the  system  remains 
a  costly  and  lengthy  manual  process.  This  alternative  would  aim  therefore  at 
implementing  a  more  cost-effective  automated  approach  to  building  knowledge 
bases  for  this  existing  prototype  system. 

These  three  alternatives  for  pursuing  the  concept  of  a  class-generic 
automated  knowledge  acquisition  aid  represent  a  range  of  tradeoffs  among 
issues  involving  cost,  payoff,  and  feasibility.  The  tradeoffs  cannot  be 
resolved  on  purely  technical  grounds  in  favor  of  any  one  of  the  al ternatives. 
Instead,  Navy  priorities  will  need  to  be  considered  in  selecting  an 
alternative  for  further  development. 

Analysis  of  the  knowledge  engineering  process  and  human  factors 
considerations  led  to  a  set  of  guidelines  for  the  development  of  user 
interfaces  to  these  systems.  These  guidelines  emphasize  system 
characteristics  required  to  insure  the  utility  of  the  knowledge  acquisition 
aid  to  Navy  domain  experts. 

Discussions  of  user  interface  design  issues  for  the  class-generic 
knowledge  acquisition  aid  were  held  with  several  Navy  domain  experts  and 
training  system  developers.  These  discussions  provide  some  guidance  for  the 
pursuit  of  any  of  the  alternatives.  The  most  significant  and  consistent 
opinion  expressed  Indicated  that  the  medium  of  human-machine  interaction— 
frequently  the  focus  of  so-called  "user-friendly"  interface  design— is  not 
likely  to  be  the  crucial  factor  in  determining  the  utility  of  a  knowledge 
acquisition  system  to  Navy  domain  experts.  Rather,  discussants  emphasized  the 
need  for  conceptual  support  for  the  knowledge  specification  task  in  the  user 
interface  ah(T  the"  importance  of  selecting  potential  users  with  some  computer 
skills  and  high  motivation  to  use  the  system.  They  perceived  high  motivation 
to  contribute  to  a  system  development  effort  as  the  sine  qua  non  for  effective 
automated  knowledge  acquisition:  the  best  user  interface  would"  not  be 
sufficient  to  support  use  by  domain  experts  arbitrarily  assigned  to  work  with 
a  system  development  team.  Given  high  motivation  and  some  computer  skills 
(which  are  increasingly  widespread  among  Navy  personnel),  conceptual  support 
from  the  user  interface  becomes  the  most  critical  design  issue. 

An  interface  to  a  knowledge  acquisition  system  should  provide  several 
types  of  conceptual  support.  These  Include  (1)  adaptive  control  of  dialogue 
initiative,  (2)  user  access  to  information  about  prior  and  potential  future 
contexts  for  his  activities  with  the  system,  and  (3)  feedback  about  how 
knowledge  supplied  by  the  user  affects  the  behavior  of  the  training  system. 
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The  architecture  of  the  knowledge  acquisition  system  and  Its  Interface  should 
therefore  Include  intelligent  dialogue  planning  and  interaction  management 
functions.  Further,  it  should  provide  a  flexible  Interface  to  the  target 
training  system  for  experimentation  with  the  incrementally  developed  knowledge 
base  supplied  by  the  user.  Detailed  design  decisions  regarding  Interaction 
media  and  protocols  can  only  be  resolved  when  the  characteristics  of  the 
domain  class  and  features  of  the  training  system's  own  architecture  have  been 
identified. 

Two  activities  therefore  emerge  as  critical  steps  toward  the  development 
and  implementation  of  any  of  the  three  alternative  knowledge  acquisition 
concepts.  First,  research  must  focus  on  the  problem  of  characterizing  and 
representing  class-generic  knowledge  necessary  to  capture  training  expertise 
in  a  variety  of  domains.  Second,  techniques  must  be  devised  to  use  this 
knowledge  in  an  appropriate  system  architecture  providing  conceptual  support 
for  the  user  during  knowledge  acquisition  sessions.  Further  design  of  low- 
level  interface  details  can  be  deferred  at  least  until  a  prototype  system 
accessible  to  motivated,  skilled  users  has  been  developed. 
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PREFACE 


The  application  of  knowledge-based  modeling  techniques  to  training 
simulators  appears  likely  because  of  increasing  pressures  to  make  these 
devices  more  "intelligent."  There  is  an  emerging  requirement  to  decrease  the 
instructor/student  ratio  for  simulator-based  training.  This  requirement  is 
dictated  by  logistical  considerations.  In  order  to  counter  this  potential 
threat  to  training  effectiveness,  intelligent  training  devices  implemented 
through  knowledge-based  models  could  be  developed  to  augment  the  instructor 
cadre.  Instructor  functions  would  be  provided  in  the  form  of  software  models. 
This  will  allow  the  instructor/student  ratio  to  be  reduced  without  reducing 
the  amount  or  quality  of  instructional  guidance. 

The  Human  Factors  Laboratory  at  the  Naval  Training  Equipment  Center  has 
been  pursuing  research  related  to  the  development  of  intelligent  training 
devices  for  over  a  decade  through  research  programs  in  adaptive  training, 
student  performance  measurement,  and  part-task  training.  It  is  becoming 
apparent  that  such  functions  can  most  effectively  be  implemented  using 
concepts  borrowed  from  the  artificial  intelligence  research  community. 
Attention  is  being  focused  on  knowledge-based  models,  in  particular,  expert 
systems. 

The  current  development  cycle  of  an  expert  system  is  very  lengthy  and 
consumes  a  great  deal  of  resources.  Making  the  implementation  process  more 
efficient  would  support  the  application  of  these  knowledge-based  models  to 
simulator-based  training.  In  particular,  the  knowledge  engineering  process 
has  to  be  streamlined.  The  present  task  was  initiated  to  investigate  the 
possibilities  for  reducing  resource  expenditures  during  the  process  of 
knowledge  engineering.  The  stated  goals  were  to  determine  the  extent  to  which 
the  process  could  be  automated  and  to  make  recommendations  concerning  the 
conditions  under  which  such  automation  would  be  practical.  Clearly,  knowledge 
engineering  is  a  complex  cognitive  activity  and  a  general,  completely 
automated  procedure  cannot  be  supported  by  the  current  state  of  technology. 
However,  it  appears  that  a  workable  system  can  be  developed  to  automate 
certain  phases  of  the  knowledge  engineering  cycle  for  particular  classes  of 
expert  models.  This  report  details  the  procedures  followed  in  reaching  this 
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SECTION  I 
INTRODUCTION 


BACKGROUND 

Much  training  in  the  Navy  and  other  military  services  requires  closely 
supervised,  intensive,  and  highly  technical  instruction  on  a  complex  task.  In 
many  cases,  such  instruction  is  provided  by  a  training  specialist  in 
conjunction  with  a  training  simulator.  Simulators  are  typically  designed  to 
develop  and  extend  knowledge  and  skills  that  are  impractical,  expensive,  or 
impossible  to  exercise  within  operational  environments.  Typically,  such 
simulators  provide  practice  on  the  target  task  but  little  or  no  instructional 
feedback  on  trainee  performance,  skill  deficiencies,  or  coaching  on  correct 
behavior.  Human  instructors  must  therefore  observe  trainee  performance  and 
provide  appropriate  instructional  interventions.  However,  in  many  cases, 
skilled  Navy  instructors  are  in  short  supply  relative  to  the  number  of 
trainees,  and  the  quality  of  Instruction  can  vary  from  individual  to 
individual . 

Recent  work  on  "intelligent"  simulators  is  leading  to  the  development  of 
simulators  with  instructional  capabilities  in  addition  to  simulation  of 
operational  equipment  and  situations.  These  training  simulators  will  embody 
"surrogate  instructors",  which,  in  conjunction  with  human  instructors,  could 
better  provide  trainee  performance  evaluation  and  adaptive  training.  Such 
augmented  training  capabilities  can  significantly  increase  the  cost- 
effectiveness  of  simulator-based  training  and  extend  the  availability  of 
individualized  instruction. 

Research  on  surrogate  instructor  technology  (also  called  Intelligent 
Computer-Assisted  Instruction  [ICAI])  has  utilized  artificial  intelligence 
techniques  to  represent  conceptual  knowledge  about  the  problem  domain,  expert 
and  trainee  performance,  and  instructional  methods.  Many  of  these  techniques 
are  derived  from  those  used  in  so-called  "expert  systems"--knowledge- 
intensive,  high-performance  programs  designed  to  serve  as  automated 
consultants  to  domain  experts.  A  recognized  bottleneck  in  the  development  of 
expert  systems  and  hence  surrogate  instructor  systems  is  the  human  resources, 
time,  and  cost  required  to  articulate  the  expert  domain  knowledge  and  to 
encode  it  in  software.  Current  approaches  involve  frequent,  long-term 
interactions  among  a  team  of  highly-trained  knowledge  engineers  and  domain 
experts.  While  some  technology  has  been  developed  to  assist  knowledge 
engineers  in  developing  expert  consultation  systems,  it  has  not  reduced 
requirements  for  person-to-person  Interactions  that  account  for  much  of  the 
development  time  and  cost.  The  application  of  technology  assistance  to 
developing  surrogate  instructor  systems  has  lagged  behind  expert  systems 
development,  and  little  work  has  addressed  the  requirements  for  surrogate 
instructional  systems  insofar  as  they  differ  from  expert  consultation  systems. 
Therefore,  a  need  exists  to  Identify  technological  opportunities  to  facilitate 
and  streamline  the  task  of  articulating  expert  knowledge  to  be  used  in  the 
development  of  an  automated  Instructional  system. 
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APPROACH 

The  current  project  sought  to  define  a  set  of  feasible,  high-payoff, 
research  objectives  for  the  automation  of  the  knowledge  elicitation  process. 

As  a  first  step,  a  review  was  made  of  the  available  documentation  of  previous 
and  current  expert  and  instructional  system  building  efforts.  Particular 
attention  was  given  to  those  projects  attempting  to  develop  tools  to  aid  the 
system  building  process  and  to  provide  generic  system  capabilities. 
Conversations  with  other  knowledge  engineers  provided  insights  into  the 
difficulties  and  pitfalls  of  designing  expert  systems.  In  addition,  these 
individuals  provided  useful  comments  on  the  design  concepts  developed  to  meet 
NAVTRAEQUIPCEN  requirements.  Finally,  interviews  with  domain  experts  and 
training  specialists  in  the  Navy  elucidated  system  design  constraints  dictated 
by  characteristics  of  target  users  and  the  operational  environment. 

Early  on  in  the  review  and  analysis  effort,  it  became  apparent  that  the 
present  state-of- the-art  in  the  field  of  expert  systems  can  not  support  a 
detailed  generic  design  for  surrogate  instructor  systems.  Variations  across 
domains  and  desired  system  capabilities  require  different  representations  for 
knowledge  and  different  mechanisms  for  applying  it.  As  of  now,  no  single, 
uniform  representation  of  concepts,  relations,  procedures,  and  strategies  has 
been  found  sufficient  to  capture  domain  expertise  in  a  wide  variety  of 
domains. 

In  addition,  it  became  clear  that  the  software  modules  in  a  surrogate 
instructor  system  that  use  the  domain  knowledge  can  not  be  independent  of  the 
’-^presentations  or  t heir  use.  This  dependence  also  extends  to  software  that 
would  aid  the  development  of  surrogate  instructor  systems.  In  particular, 
details  of  an  effective  user-interface  design  for  acquiring  domain  knowledge 
from  an  expert  depend  on  the  knowledge  representation.  Generic  human 
engineering  principles  for  interface  design  are  only  rough  guidelines  to 
system  development,  and  their  application  requires  more  detailed 
interpretation  with  respect  to  the  target  system's  specific  features  and 
implementation.  Thus,  we  concluded  that  an  intelligent,  generic  system 
intended  to  support  the  development  of  any  surrogate  instructor  system  was  not 
feasible  given  the  current  technology  and  state  of  knowledge  in  expert 
systems. 

We  therefore  worked  to  determine  a  more  restricted  concept  of  generality 
f,>r  .in  automated  knowledge  acquisition  system.  Using  this  concept,  we 
attempted  to  elucidate  (1)  a  design  for  a  knowledge  acquisition  system 
specified  t.o  the  extent  possible  without  coinnlttlng  to  a  particular  surrogate 
instructor  system  architecture,  (2)  a  set  of  alternative  interface  concepts 
consistent  with  the  design  for  which  additional  work  could  be  realistically 
undertaken  to  produce  a  useful  system,  and  (3)  a  set  of  issues  that  must  be 
resolved  in  such  additional  work. 

The  concept  we  developed  addresses  generality  for  an  automated  knowledge 
acquisition  system  at  the  level  of  a  cl  ass  of  tasks,  each  member  of  which  can 
r>e  adequately  served  by  a  fixed  set  of  surrogate  instructor  capabilities  and 
knowledge  representation  formalisms.  The  notion  of  such  classes  is  largely 
intuitive:  a  characteristic  of  members  within  a  class  is  congruence  of  high- 
level  semantic  and  pragmatic  aspects  of  domain  knowledge,  whicn  we  will  refer 
to  as  class-generic  knowledge.  However,  regardless  of  its  Intuitive  nature, 
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In  our  work  we  have  found  that  consensus  exists  for  the  definition  of  some 
such  classes  of  tasks. 

This  report  presents  the  results  of  our  research  to  specify  and  elaborate 
a  design  for  an  Instructional  knowledge  acquisition  system.  Section 
II  presents  a  model  of  the  knowledge  engineering  process  and  describes  the 
activities  required  to  build  an  intelligent  instructional  system.  This 
section  indicates  how  particular  features  of  that  process  preclude  the  full 
generality  NAVTRAEQUIPCEN  sought  in  its  original  concept.  Section  III  reviews 
selected  research  on  tools  to  assist  expert  system  building.  This  review 
illustrates  more  concretely  the  factors  that  limit  generality.  Section 
IV  presents  our  general  system  concept,  and  a  set  of  criteria  to  be  considered 
in  evaluating  alternative  realizations  of  the  concept.  Three  alternative 
specific  concepts  are  then  presented  that  differ  with  respect  to  these 
criteria  and  to  the  specific  instructional  capabilities  of  the  systems  they 
serve.  Section  V  introduces  a  set  of  user  interface  design  Issues  and 
describes  interviews  conducted  with  system  builders  and  domain  experts  to 
determine  how  those  design  Issues  might  be  resolved  in  realizing  our  system 
concept.  Section  VI  presents  an  architecture  for  implementing  those  functions 
and  interface  features  of  the  system  concept  that  can  be  specified  without 
further  comnitment  to  the  class  of  tasks  and  the  host  system's  capabilities. 
Finally,  Section  VII  reviews  the  conclusions  and  recommendations  derived  from 
the  research. 
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SECTION  II 

THE  KNOWLEDGE  ENGINEERING  TASK 


Planning  effective  automation  for  aspects  of  any  complex  task  requires  an 
analysis  of  the  task  and  development  of  a  process  model  of  the  task.  This 
model  is  required  both  for  determining  what  task  components  to  automate  and 
how  that  automated  system  should  Interface  to  the  people  who  will  use  it. 

THTs  section  presents  an  analysis  of  the  knowledge  engineering  task  and  a 
model  that  describes  the  relationships  among  task  objectives,  the  activities 
that  attain  them,  and  the  knowledge  required  by  those  activities. 

WHAT  IS  KNOWLEDGE  ENGINEERING? 

Knowledge  engineering  is  the  process  by  which  a  class  of  computer 
programs  called  expert  systems  are  created.  These  systems  are  built  to  aid  or 
perform  tasks  that  are  very  knowledge  intensive,  typically  require  inexact  and 
imprecise  reasoning,  and  for  which  the  expertise  for  performing  the  task 
resides  primarily  with  a  very  few  human  "experts."  These  tasks  involve  some 
form  of  situation  interpretation,  decisionmaking,  and/or  planning.  The  best- 
known  examples  of  recent  expert  systems  include  programs  for  medical  diagnosis 
(Shortliffe,  1976),  chemical  analysis  (Lindsay,  et  al_. ,  1980),  geological 
analysis  (Duda,  et  al.,  1978),  and  planning  computer  system  configurations 
(McDermott,  1980T.  Knowledge  engineering  differs  from  conventional  software 
engineering  in  (1)  the  nature  and  extent  of  the  interaction  between  the 
knowledge  engineer! s)  (KE)  and  the  domain  expert! s)  (DE),  and  (2)  the  types  of 
software  design  and  implementation  tools  usea. 

The  KE  is  more  dependent  on  the  DE  both  prior  to  implementation  and  for 
cost- implementation  testing  than  is  typical  in  other  software  engineering 
efforts.  Because  expert  systems  are  knowledge  intensive,  knowledge 
elicitation  is  one  of  the  KE ’ s  major  technical  objectives.  It  alone  generates 
a  need  for  frequent  and  prolonged  interaction  between  the  KE  and  DE  prior  to 
and  during  Implementation. 

Knowledge  acquisition  has  proved  to  be  a  complex  and  difficult  process. 
The  knowledge  that  must  be  incorporated  into  an  expert  system  is  largely  non- 
numerical  and  imprecise.  It  is  usually  expressed  as  an  extensive  body  of 
concepts,  rules,  and  approximate  methods.  The  term  heuristic  is  used  to 
describe  both  this  type  of  expert  knowledge  and  the  type  of  programming  a  KE 
uses  to  operationalize  it.  The  expert's  heuristic  knowledge  is  often  tacit 
and  thus  difficult  both  to  elicit  and  articulate.  Determining  whether  the 
evolving  knowledge  base  is  consistent  and  when  it  is  complete  enough  is 
therefore  a  major  problem  for  the  KE  and  DE.  This  fact  and  the  imprecision  of 
heuristic  knowledge  increase  the  KE's  dependence  on  the  DE  for  debugging, 
evaluating,  and  tuning  performance  of  an  expert  system.  Thus,  knowledge 
acquisition  typically  continues  well  into  the  Implementation  stage. 

The  data  types  and  control  structures  of  conventional  programming 
formalisms  are  not  conceptually  well -suited  for  describing  heuristic  knowledge 
and  for  developing  expert  systems.  They  do  not  provide  an  organizational 
framework  for  knowledge  acquisition  and  lack  good  facilities  for  debugging 
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knowledge-intensive  code.  Research  on  expert  systems  has  therefore  evolved 
representation  languages,  symbolic  programing  languages,  and  rule-based 
problem-solving  architectures  to  allow  more  intuitive  and  transparent 
operational ization  of  heuristic  knowledge  on  computers.  In  addition, 
specialized  system-building  tools  have  been  developed  (e.g.,  Davis  [1977], 
Reboh  [1981])  to  aid  the  KE  in  encoding  knowledge  into  specific 
representations  and  in  testing  and  refining  an  implementation.  (Section 
III  reviews  these  design  and  Implementation  tools.)  These  tools  are  designed 
to  be  used  interactively;  hence,  they  enable  and  support  the  incremental 
system  building  and  testing  necessary  in  expert  systems. 

THE  PRACTICE  OF  KNOWLEDGE  ENGINEERING 

As  a  research  area,  knowledge  engineering--a  sub-field  of  artificial 
intelligence  ( AI )  —  is  about  15  years  old.  The  total  number  of  practitioners 
is  under  300  internationally,  mostly  located  in  universities.  Only  very 
recently  have  serious  applied  and  commercial  development  efforts  outside 
academic  research  centers  been  undertaken.  The  academic  focus  has  been  on  the 
individualistic,  innovative,  and  high-risk  features  of  research  in  an  emerging 
field  rather  than  on  systematically  reducing  to  practice  the  process  of 
knowledge  engineering.  As  a  result,  the  construction  of  expert  systems  is 
still  more  of  an  art  or  craft  than  a  discipline. 

Knowledge  engineering  is  a  highly  Intellectual  and  individualistic 
process,  although  it  is  not  uncommon  for  several  KEs  to  work  together  on  a 
project.  KEs  generally  hold  graduate  degrees  either  in  computer  science  or 
cognitive  psychology  and  are  familiar  with  the  concepts  and  methods  of  AI. 
However,  they  receive  little  formal  training  on  how  to  build  expert  systems. 
Instead,  they  acquire  the  necessary  skills,  usually  as  graduate  students,  in  a 
loose  apprenticeship  system  under  the  supervision  of  more  experienced 
practitioners.  As  in  any  apprenticeship  program  there  is  considerable 
variability  in  supervision.  Progress  in  learning  knowledge  engineering 
through  hands-on  experience  is  Impeded  by  a  lack  of  precise  criteria  for 
assessing  a  KE's  performance.  Furthermore,  it  is  not  obvious  at  what  point 
someone  becomes  a  qualified  knowledge  engineer. 

Without  exception,  documentation  of  existing  knowledge  engineering 
efforts  describes  numerous  false  starts  and  revision  cycles  prior  to  achieving 
a  system  of  any  practical  value.  This  characteristic  may  reflect  in  part  the 
academic  research  setting  of  the  efforts  and  its  tendency  to  encourage 
inventiveness  and  discovery  of  alternatives  even  after  existing  efforts  have 
demonstrated  workable  methods  and  tools.  However,  there  are  substantive 
limitations  on  the  process  of  building  expert  systems  that  are  more  important 
for  explaining  its  characteristics.  First,  there  are  problems  in  formulating 
initial  system  specifications;  existing  experience  is  either  too  limited  or 


^Arguably,  the  same  remark  could  be  made  about  other  software  engineering  as 
well.  However,  this  field  is  more  mature  and  considerable  effort  during  the 
past  decade  has  attempted  to  organize  and  standardize  the  production  of 
commercial  software. 
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has  not  been  sufficiently  analyzed  to  determine  exactly  what  an  effective 
expert  system  for  a  task  domain  should  do.  Second,  there  are  problems  in 
implementing  system  specifications;  there  is  a  similar  lack  of  experience  or 
understanding  regarding  the  relationship  between  a  set  of  specifications  and 
appropriate  methods  and  tools  for  achieving  it.  At  least  at  this  point  in 
time,  the  knowledge  available  to  the  KE  from  prior  efforts  for  generating  and 
evaluating  designs  Is  heuristic— as  heuristic  as  the  domain  knowledge  he 
himself  must  acquire  and  operationalize  to  build  an  expert  system.  This  use 
of  heuristic  knowledge  by  the  KE  accounts  in  part  for  the  iterative  nature  of 
the  knowledge  engineering  process.  It  also  suggests  that  analyzing  and 
modeling  what  the  KE  does,  in  order  to  consider  how  to  automate  aspects  of 
knowledge  engineering,  may  Itself  be  viewed  as  an  exercise  in  knowledge 
engineering.  However,  the  scope  of  the  knowledge  engineering  task  is 
considerably  broader  than  that  of  any  task  for  which  an  expert  system  has  yet 
been  built.  Modeling  the  knowledge  engineering  task  is  therefore  important  to 
permit  identification  of  smaller,  relatively  Independent  activities  that  might 
be  feasible  candidates  for  automation. 

A  COGNITIVE  TASK  MODEL  OF  KNOWLEDGE  ENGINEERING 

The  design  of  automated  knowledge  engineering  functions  first  requires  a 
cognitive  task  model  specifying  the  task's  activities  and  the  relationships 
among  them.  The  naive  approach  to  introducing  automation  suggests  that  where 
implementation  is  technically  feasible  and  cost-effective,  the  machine  should 
perform  all  activities  in  which  Its  productivity  exceeds  that  of  the  human. 
This  approach  overlooks  the  need  to  consider  the  nature  of  the  human-machine 
interaction  that  must  occur.  When  activities  are  divided,  those  that  are 
retained  by  the  human  may  include  some  that  depend  on  knowledge  and 
information  generated  by  activities  assigned  to  the  machine.  Likewise,  the 
machine  may  be  dependent  on  knowledge  generated  from  the  human's  activities. 

A  cognitive  task  model  specifies  how  tasks  depend  on  knowledge  and  information 
generated  or  modified  by  other  tasks.  Thus,  the  model  can  be  used  to 
determine  what  knowledge  must  pass  the  interface  between  human  and  machine  and 
the  frequency  with  which  that  interface  must  be  used.  It  can  show  that 
although  some  tasks  can  be  performed  better  in  isolation  by  the  machine,  the 
cost-effectiveness  or  viability  of  the  overall  system  requires  that  those 
activities  be  retained  by  the  human.  Although  human  performance  is  remarkably 
flexible,  ill-considered  human-machine  architectures  can  overwhelm  human 
cognitive  capacity  and  endurance  or  impair  motivation  for  using  the  system. 

A  cognitive  task  model  is  therefore  an  Important  tool  in  determining  both 
what  task  components  to  automate  and  how  to  design  the  human-machine 
Interface.  It  enables  an  understanding  of  how  the  attainment  of  objectives  is 
shaped  by  requirements  for  knowledge,  performance  factors  and  external 
constraints.  This  understanding  permits  a  design  for  automation  that  can 
enable  a  human-machine  "team"  to  to  attain  the  task  objectives  more 
effectively  or  efficiently  than  can  a  human  working  alone.  The  model  of  the 


'’it  might  also  lead  to  a  conclusion  for  some  tasks  that  automation  woulJ  not 
be  feasible  or  cost-effective. 
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knowledge  engineering  task  postulates  activities  performed  by  three 
participants:  the  customer  (the  ultimate  end  user  and  perhaps  the  financial 
sponsor  of  the  knowledge  engineering  product),  the  knowledge  engineer  (the 
designer  and  implementor  of  the  system),  and  the  domain  expert  (the  source  of 
domain-speci fic  knowledge). 

Figure  1  sumnarizes  the  knowledge  engineering  process  as  the  interacting 
objectives  and  activities  of  these  participants.  It  presents  the  set  of  tasks 
as  a  flow  chart  indicating  the  ordering  of  and  dependencies  among  the  various 
activities  involved  in  a  knowledge  engineering  effort.  Because  we  have 
summarized  the  process  in  flow  chart  form,  the  objectives  and  tasks  may  appear 
to  have  a  "natural"  linear  order.  However,  this  appearance  is  deceiving,  for 
it  presupposes  a  mature  design  science  for  building  expert  systems.  The 
ability  to  design  a  system  successfully  (i.e.,  so  that  the  first 
implementation  of  the  design  performs  acceptably)  requires  (1)  a  set  of 
general  design  principles,  (2)  knowledge  required  to  apply  the  design 
principles  to  the  particular  problem  at  hand,  and  (3)  a  method  to  determine 
that  the  resulting  design  is  complete  and  satisfactory.  In  the  knowledge 
engineering  process,  there  is  no  comprehensive  or  generally  accepted  body  of 
knowledge  for  meeting  any  of  these  requirements.  Therefore,  achieving  a 
workable  design  most  typically  requires  the  interaction  among  component 
processes  of  design,  implementation,  test,  refinement,  and  redesign.  No  one 
of  these  processes  proceeds  in  isolation;  rather,  several  are  simultaneously 
active  and  under  consideration.  Thus,  a  linear  stage  model  has  inherent 
limitations  as  a  description  of  this  complex  process. 

An  alternative  to  the  linear  mode)  is  a  class  of  models  that  accommodates 
the  simultaneous  operation  of  multiple  cooperating  processes.  These  models, 
called  blackboard  models,  have  been  used  to  model  other  complex  cognitive 
processed  sucTT  as  pTanTfTng  (Hayes-Roth,  1980;  Thorndyke,  McArthur,  and 
Camnarata,  1981),  decision  making  (Thorndyke,  1982),  speech  understanding 
(Erman,  et  al_. ,  1980),  reading  (Rumelhart,  1976),  and  sensor  interpretation 
(Nii,  et:  aU ,  1982).  We  believe  that  an  attempt  to  develop  a  detailed 
computational  model  of  the  knowledge  engineering  task  might  profitably  adopt 
this  modeling  approach,  since  the  non-deterministic  order  of  activities  is 
easily  accommodated  by  this  framework.  However,  one  limitation  of  blackboard 
models  is  the  difficulty  of  representing  and  illustrating  succinctly  the 
individual  activities  required  for  task  performance  or  the  relationships  among 
them.  In  this  respect,  the  linear  model  is  superior  in  its  ease  of 
illustration. 

Since  a  clear  explication  of  the  knowledge  engineering  process  is 
fundamental  to  the  subsequent  understanding  of  recommended  proposals  for 
automation  of  components  of  that  process,  we  have  chosen  to  Illustrate  the 
knowledge  engineering  process  as  a  linear  model,  as  shown  in  Figure  1. 

However,  we  recognize  that  the  nominal  sequential  order  implied  by  the  figure 
is  not  a  strictly  accurate  characterization  of  the  way  in  which  knowledge 
'’ngineering  proceeds.  In  recognition  of  the  non-determinism  in  the  order  of 
activities,  we  have  used  an  unconventional  notation  in  the  flow  chart  shown  in 
Figure  1.  Several  of  the  branches  leading  from  decision  boxes  lead  to 
multiple  points.  This  notation  indicates  that  any  one  of  the  indicated 
activities  can  be  undertaken  next  depending  on  conditions  not  represented  in 
the  flow  diagram. 
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Strictly  speaKing,  the  KE's  primary  objective  is  to  create  an  operational 
expert  system  that  solves  a  problem  for  his  customer.  Six  major  technical 
objectives  must  be  attained  to  create  an  operational  expert  system: 


a.  Definition:  Specification  of  the  capabilities  the  system  will 
exhibi t. 

b.  Knowledge  Acquisition:  Description  of  the  knowledge — concepts, 
facts,  an<f  prom em  sol vfng  methods — needed  to  achieve  the  specified 
capabi 1 ities. 

c.  Formalization:  Organization  of  the  knowledge  using  formal 
representation  Tang’uages. 

d.  Design:  Determination  of  hardware  and  software  archi tecture. 

e.  Implementation:  Implementation  of  the  encoded  knowledge, 
procedure’s,  ancTuser  interface. 

f.  Testing:  Evaluation  and  refinement  of  the  system. 

These  six  objectives  are  similar  to  those  described  elsewhere  (see,  for 
example,  Buchanan,  et  al.  [1983]  and  Reboh  [1981]).  In  examining  how  the  six 
objectives  have  been  acfifressed  in  prior  systems,  it  is  important  to  note  that 
most  previously  developed  expert  systems  are  products  of  R&D  environments.  In 
these  efforts,  creating  a  working  system  was  often  secondary  to  performing 
successful  science  (e.g.,  the  development  of  new  knowledge  representations, 
problem-solving  methods  or  system-building  tools).  This  difference  in  the 
primary  motivation  of  RAD  and  of  potential  product-oriented  applications  is 
important  because  it  leads  to  different  subobjectives  for  the  six  major 
technical  objectives  and  entails  different  external  constraints  on  the  KE. 

For  example,  in  academic  research  the  objective  of  Formal ization  may  entail 
inventing  a  new  representation .  In  an  applications  effort,  on  the  other  hand, 
external  constraints  may  strongly  discourage  the  KE  from  inventiveness  in 
favor  of  selection  from  existing  representation  languages.  In  fact,  as  we 
will  explain  later,  he  may  even  be  constrained  a  qriori  to  use  a  particular 
representati on  language.  Thus,  in  developing  a  task  model  based  on  prior  case 
studies,  we  took  into  account  how  academic  research  motivations  affected 
pursuit  of  the  objective  of  building  a  working  system.  By  so  doing,  we  have 
oriented  the  model  toward  use  in  designing  expert  systems  in  product-or iented 
environments. 

The  remainder  of  this  section  discusses  the  model  in  detail.  We 
structure  our  discussion  around  the  KE's  six  technical  objectives  listed 
above.  For  each,  we  first  present  its  immediate  subobjectives  and  then 
consider  in  detail 


-  the  knowledge  the  KE  brings  to  the  task  for  attaining  that  objective 

-  its  dependencies  on  both  other  technical  objectives  and  external 
constrai nts 
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-  the  types  of  activities  that  achieve  it 

-  the  criteria  hy  which  the  KE  monitors  progress  on  it. 

A .  Def i n it  ion  Objective 

Subobject i ves 

Generate  potential  system  capabilities 
Identify  customer  and  DE  problems 
Identify  opportunities  for  technological  enhancement 
Identify  user  interface  requirements  and  constraints 

Select  target  capabilities 

Identify  cost-effective  candidates 
Identify  technologically  feasible  candidates 
Determine  cognitively  feasible  capabilities 

The  Definition  objective  entails  determining  the  performance  capabilities 
of  the  expert  system  and  how  it  will  interact  with  users.  A  planning  or 
decisionmaking  task,  for  example,  might  require  information  interpretation, 
option  generation,  option  evaluation,  and  option  selection  activities.  An 
expert  system  might  assist  the  user  with  any  of  these,  depending  on  the 
approach  to  cooperative  man-machine  problem-solving  the  customer  desires.  In 
addition,  cost-effectiveness,  feasibility,  and  other  constraints  on  the 
expected  use  of  the  system  will  influence  which  particular  capabilities  are 
selected. 

The  major  subobjectives  of  capabilities  definition  are  (a)  generation  of 
candidate  capabilities  and  (b)  filtering  those  candidates  to  produce  a  target 
set  satisfying  the  various  criteria.  The  system's  desired  capabilities  are 
initially  motivated  by  problems  in  performing  the  task.  Ordinarily,  these  are 
apparent  and  have  motivated  the  customer's  initial  decision  to  consider  an 
expert  system.  They  may  be  documented  in  written  materials  hut  can  be  most 
clearly  defined  through  interactions  between  the  KE,  the  customer  and  OF. 
Typically,  the  improvement  sought  in  the  expert  system  entails  lowering  the 
cost,  increasing  the  speed  or  reliability,  and/or  improving  the  accuracy  of 
tack  performance  relative  to  the  current  methods. 

for  the  KE  to  comprehend  the  difficulties  the  customer  and  DE  perceive  in 
their  task--and  to  evaluate  differences  in  the  perceptions  of  the  customer  and 
the  l)E  -he  needs  some  knowledge  of  the  task  and  how  it  is  performed.  If  the 
KE  is  initially  ignorant  about  the  task  domain,  his  initial  interactions  with 
the  customer  and  DE  must  provide  an  introduction  to  the  domain.  This 
introduction  explores  the  general  objectives  and  methods  of  the  task  as  it  is 
currently  performed.  The  KE  develops  the  foundations  of  a  task  model  using 


^We  distinguish  the  customer,  or  sponsor,  of  the  effort  from  the  DE  for  the 
sake  of  generality.  In  particular  cases,  the  customer  and  the  DE  may  be  the 
sane  individual,  or  the  DE  may  represent  the  interests  of  the  customer  to  the 
KF. 
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his  knowledge  of  modeling  frameworks.  As  he  adopts  a  general  frame-work,  he 
can  take  some  initiative  in  seeking  additional  information  he  needs.  Much  of 
the  domain  knowledge  the  KE  elicits  in  these  initial  interactions  will  not  be 
encoded  in  the  ultimate  system. 

Once  the  KE  has  a  basic  understanding  of  the  task  domain,  he  can  guide 
the  exploration  of  the  problems  the  customer  anH  DE  perceive  and  characterize 
their  performance  and  cost  attributes.  Initiative  by  the  KE  is  important  at 
this  point  to  foe  us  the  interactions.  Together,  the  Kf  ,  customer,  and  !)f  can 
-hen  arrive  at  .»  set  of  performs  e  capo:  ilities  that  would  remedy  the 
identified  problems. 


The  customer  and  E'f  must;  also  provide  information  about  the  intended 
users  of  the  system  and  the  environment  in  which  it  will  be  used.  This 
knowledge  is  needed  by  the  KT  to  determine  alternative  user  interface 
capabilities  for  the  system.  The  prior  experience  of  the  users  with 
interactive  computing  systems,  their  motivation  and  ability  to  use  the  system, 
and  their  technical  knowledge  of  the  domain — if  it  is  different  from  the 
DE's--are  all  considerations  in  designing  the  user  interface.  For  instance, 
interface  capabilities  that  depend  on  typed  input  can  be  i nappropriate  if 
users  cannot  be  motivated  to  learn  or  use  typing  skills  in  their  work 
environment.  This  particular  fact  has  been  learned  too  late  by  designers  of 
some  commercial  software  systems  oriented  tGward  managers. 

Another  type  of  constraint  that  may  influence  system  capabilities  and 
interf ace  design  involves  hardware  or  software  requirements.  The  customer'  may 
have  cost  considerations,  existing  equipment.,  or  organizational  policies  that 
dictate  the  use  of  a  particular  system.  If  so,  the  capabilities  provided  by 
this  system  may  either  restrict  or  present  opportunities  for  the  use  of 
certain  interface  media,  software  packages,  and  orogramming  languages. 

Once  he  has  an  under  x  tan  a ;  ug  ->f  the  domain  md  the  constraints  on  sv  x  ton 
development,  the  KC  can  genera  In  additional  and i date  capabilities  for  tin- 
system  based  on  technological  opportunity .  Bv  recogni  ;i ng  that  the  task 
shares  features  with  others  Tor  which  expert  systems  have  been  built,  the  E 
may  propose  to  include  capabilities  present  in  those  systems.  The  decision  to 
include  such  capabilities  may  involve  considerations  not  apparent  to  the 
customer  and  the  DF .  lor  example,  the  decision  say  be  based  on  the  KE's 
knowledge  that  the  capability  is  an  easy  enhancement--one  which  may  be 
virtually  "free"  because  the  knowledge  it  requires  will  already  be  used  to 
implement,  another  capability.  The  KE's  ability  to  realize  technological 
opportunity  in  generating  capabilities  is  dependent  on  his  knowledge  of  prior 
expert  systems  and  his  ability  to  draw  analogies  from  features  of  those 
systems  to  the  cur-rent  task. 

The  second  objective  in  capabilities  definition  i‘  filtering  the 
candidate  capabilities  to  produce  a  target  set.  The  primary  selection 
'riferir  are  the  cost -benefi f  expectation  for  a  capability,  the  technological 
feasibility  of  a  capability,  and  the  overall  operational  coherence  of  the 
target  set.  Tost -henet i t  evaluation  depends  on  (T  )  the  resources  the  customer 
is  willing  to  allocate  for  developing  and  operating  an  expert  system;  (?)  th-' 
Kl  ’  s  ability  to  estimate  the  development  effort  and  computational  resources  a 
capability  will  require;  and  (3)  the  beliefs  that  the  KE,  the  customer,  dad 
the  gr  have  about  the  value  of  enhancing  existing  task  performance. 
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Technological  feasibility  becomes  a  difficult  issue  when  there  are  no 
precedents  for  implementation  of  a  capability  or  when  there  are  initial 
constraints.  The  KE  must  estimate  the  risk  involved  in  pursuing  development 
of  the  capability  and  a  joint  decision  must  be  made  about  whether  the  risk  is 
acceptable . 

In  addition  to  evaluating  capabilities  individually,  the  coherence  of  the 
entire  operational  concept  must  be  considered.  A  piecemeal  system  with 
loosely-coupled  component  capabilities  may  be  inefficient  and  difficult  to 
use.  The  set  of  capabilities  must  be  evaluated  with  respect  to  overall 
implementation  and  support  requirements  and  to  the  task  model  the  KE  develops. 
The  model  is  used  to  consider  how  dependencies  among  candidate  capabilities 
should  determine  the  composition  of  the  target  set.  For  example,  suppose 
among  a  set  of  candidates  were  included  an  automated  deduction  capability  and 
a  hierarchical  explanation  capability.  Suppose  that  for  either  cost-benefit 
or  feasibility  reasons  a  decision  was  made  to  eliminate  or  at  least 
substantially  change  the  deduction  mechanism.  If  the  purpose  or  expected 
benefit  of  the  explanation  capability  depends  in  some  way  on  the  deduction 
capability,  then  it  too  must  be  eliminated  or  modified. 

Interface  capabilities  must  be  considered  again  at  this  point  as  well. 

The  interfaces  to  different  capabilities  must  be  coherent  with  one  another  and 
with  user  requirements.  To  the  extent  that  the  cost-effectiveness  or 
feasibility  of  specific  technical  capabilities  are  linked  to  these  interface 
capabi 1 i ties ,  the  technical  capabilities  may  themselves  need  to  be 
reconsidered.  For  example,  suppose  there  is  an  option  generation  capability 
in  which  the  user  can  supply  constraints  and  an  interface  supporting  quasi¬ 
natural  language.  The  KE  may  determine  that  the  interface  will  be 
inappropriate  for  expressing  constraints.  One  or  both  capabilities  may  then 
need  to  be  revised  or  eliminated  from  the  target  set.  In  that  case,  other 
capabilities  that  depend  on  them  nay  need  to  be  reconsidered  as  well.  Thus, 
while  individual  capabilities  may  be  filtered  for  singular  reasons,  each 
addition  or  deletion  from  the  target  set  may  entail  ramifications  for  other 
capabilities  already  included  or  excluded. 

According  to  one  source  (Buchanan,  et  aJL ,  1983),  difficulties  in 
initially  defining  system  capabilities  are  among  the  major  causes  of  the 
inefficient  process  of  iterative  development  and  revision  in  knowledge 
engineering.  A  major  contributor  to  these  difficulties  is  the  "knowledge  gap" 
between  the  KE,  the  customer,  and  the  DE.  This  gap  is  inherent  in  the 
knowledge  engineering  task,  since  a  defining  feature  of  an  expert  system  is 
its  codification  knowledge  known  only  to  a  few  specialists.  Only  some  of  the 
i)L ' s  extensive  task  knowledge  may  be  needed  to  implement  system  capabilities, 
but  almost  all  of  it  is  relevant  to  selecting  which  capabilities  to  implement. 
The  Dt  may  have  difficulty  articulating  this  knowledge,  since  it  is  tacit  and 
id  hoc.  The  KF  is  an  expert  in  modeling  and  programming  formalisms  unfamiliar 
even  to  those  who  use  computers  in  more  routine  applications  such  as  data 
processing  and  data  base  management.  The  customer  and  the  HE  may  themselves 
have  varying  perspectives  on  the  task  and  the  reasons  for  seeking  to  apply 
expert  systems  technology.  The  customer  is  typically  a  manager  who  may  not 
have  current  technical  knowledge  but  who  may  have  a  perspective  on  the 
organizational  context  surrounding  the  OE's  task.  Definition  of  capabilities 
requires  the  exchange  and  integration  of  these  different  types  of  knowledge, 
oerspectives,  and  technical  vocabularies. 
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The  difficulty  of  rapidly  bridging  the  knowledge  gap  is  thus  a 
significant  factor  mitigating  against  a  successful  one-step  effort  to  define 
system  objectives.  However,  even  if  a  single  person  qualified  as  KE  and  as  DE 
for  a  task,  other  factors  would  cause  revisions  of  initially  targeted 
capabilities.  These  factors,  to  be  elaborated  in  subsequent  discussion  of 
other  technical  objectives,  involve  the  incomplete  and  uncertain  nature  of 
current  expertise  in  knowledge  engineering.  During  the  phases  of  Knowledge 
Acquisition,  Design,  Implementation,  or  Testing,  initial  estimates  of  cost  or 
feasibility  may  have  to  be  modified,  creating  a  need  to  modify  the  targeted 
system  capabilities.  Since  Knowledge  Acquisition,  Formalization,  Design,  and 
Implementation  depend  on  the  targeted  capabilities,  changes  in  Definition  will 
then  necessitate  revision  of  efforts  in  these  phases  as  well. 

To  summarize,  Definition  is  the  most  crucial  technical  objective  in 
determining  the  course  of  a  knowledge  engineering  effort.  Our  conclusion  is 
that  several  factors  preclude  an  initial  viable  definition  of  capabilities. 
This  limitation  is  one  major  reason  that  knowledge  engineering  is  highly 
variable  and  iterative  in  the  way  it  pursues  its  objectives.  The  most  salient 
factor  is  the  knowledge  gap  among  the  participants,  which  inhibits  the 
comnuni cation  required  to  define  the  desired  expert  system.  We  see  no  purely 
technical  approach  to  reducing  the  knowledge  gap  more  rapidly.  Another 
important  factor  is  the  heuristic  nature  of  the  KE's  knowledge  for  system 
design  and  implementation,  which  may  be  inherent  or  may  simply  reflect  the 
present  immaturity  of  the  emerging  discipline  of  knowledge  engineering. 

B..  Knowledge  Acquisition  Objective 

Subobjecti ves 

Identify  knowledge  categories  required  by  targeted 

system  capabilities 

Elicit  domain  knowledge  from  DE 

Informally  structure  knowledge 

The  Knowledge  Acquisition  objective  refers  to  the  accumulation  of  the 
specific  knowledge  required  to  support  implementation  of  the  targeted  system 
capabilities.  As  discussed  above,  domain  knowledge  is  also  elicited  in 
pursuing  other  objectives  of  the  knowledge  engineering  process.  In  all  cases, 
the  activities  involved  in  collecting  domain  knowledge  may  be  nearly 
identical:  interviewing  the  DE,  observing  the  DE ,  reading  printed 
descriptions,  etc.  However,  depending  on  the  technical  objective  that  is  the 
lT  ' s  current  focus,  the  form  and  sequence  of  questions  and  the  organization  of 
answers,  observations,  and  notes  can  differ.  Roughly  speaking,  the  KE  is  more 
interested  ir  the  "what"  (the  goals)  of  the  DE's  performance  when  defining 
system  objectives  and  in  the  "how"  (the  methods)  when  pursuing  Knowledge 
Acquisition.  The  specific  knowledge  that  needs  to  be  acquired  is  determined 
by  what  capabilities  have  been  targeted.  The  KE's  ability  to  characterize  the 
nature  and  scope  of  that  knowledge  and  to  develop  a  plan  for  acquiring  it 
efficiently  depends  on  his  knowledge  of  prior  attempts  to  build  similar 
systems.  At  the  present  time,  that  knowledge  can  provide  guidance  but  not 
detailed  procedures  for  the  KE  to  follow  (see,  for  example,  Buchanan,  et 
al.  PBJ]).  T hr  prior  case  stt  lies  available  to  the  KE  are  limited  both  in 
numb  an.1  i rt  the  types  of  tasks  considered.  The  formulation  of  a  rough 
mapping  between  task  characteristics,  expert  system  capabilities,  and  the 
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knowledge  and  techniques  needed  to  achieve  those  capabilities  is  just  now 
becoming  possible  (see,  Stefik  et  a_l_.  [1983],  for  one  attempt  to  specify  this 
mapping).  Therefore,  even  the  Best-informed  KE  cannot  specify  a  priori  the 
questions  and  observations  that  will  provide  the  knowledge  needed  to  implement 
a  capability,  even  one  similar  to  that  in  another  operational  system. 

The  knowledge  gap  between  the  KE  and  DE  that  affects  Definition  also 
affects  Knowledge  Acquisition.  The  KE  initially  does  not  have  the  domain- 
specific  semantic  and  pragmatic  knowledge  needed  to  evaluate  and  organize  the 
DE's  statements  or  to  direct  interactions  into  related  or  important  areas  of 
knowledge.  The  DE  doesn't  understand  the  KE's  technical  objectives  and 
methods  and  may  have  difficulty  articulating  his  knowledge;  thus,  he  is  rarely 
in  a  position  to  assume  initiative.  Therefore,  to  manage  Knowledge 
Acquisition  interactions,  the  KE  must  depend  on  relatively  domain- independent 
syntactic  knowledge,  based  on  a  general  framework  for  viewing  tasks  of  this 
type  Te.g.,  interpretation,  diagnosis,  etc.)  and  later  based  on  the 
representation  language  he  selects. 

General  frameworks  for  expressing  task  descriptions  provide  a  few 
fundamental  concepts  to  describe  and  relate  goals,  procedures,  and  data.  They 
provide  a  simple  uniform  syntax  for  structuring  Knowledge  Acquisition 
interactions.  The  most  typical  method  is  top-down  progressive  expansion  of 
detail.  Knowledge  acquisition  concerning  goals  involves  characterizing  their 
dependencies:  relative  priorities,  enabling  relations,  and  constraining 
relations.  Such  characterizations  frequently  include  judgments  of  degree  of 
belief  or  certainty.  Procedures  are  characterized  in  terms  of  their  enabling 
and  triggering  conditions,  the  more  primitive  actions  they  integrate,  the 
resources  they  require,  and  their  effectiveness.  Data  are  characterized  by 
their  source,  their  application,  and  often  judgments  of  reliability.  In  using 
a  general  framework  for  describing  knowledge  the  KE  relies  upon  ad  hoc  natural 
language,  if-then  rules,  and  diagrams  for  representation. 

Once  the  KE  has  selected  the  representation  language(s)  the  system  will 
use  (part  of  the  Formal ization  objective),  that  language  can  be  used  to  guide 
Knowledge  Acquisition.  Representation  languages  provide  a  more  detailed 
syntax  and  general  semantics  for  describing  goals,  procedures,  and  data.  They 
allow  the  KE  to  pose  more  precise  requests  for  knowledge  than  do  general 
frameworks.  At  the  same  time,  selection  of  a  representation  is  a  commitment 
that  excludes  some  domain  knowledge  from  the  Kf  ’ s  consideration  and  usually 
. -hunges  how  the  KE  pursues  his  several  objectives,  from  that  point,  the 
activities  for  acquiring  knowledge  and  formalizing  it  are  often  tightly 
coupled,  especially  when  the  KE  can  also  implement  the  knowledge  base 
incremental ly.  Even  with  this  coupling  however,  knowledge  acquisition  does 
not  become  a  passive,  mechanical  process  for  the  KE;  active  interpretation  and 
integration  are  still  required  (Buchanan,  1981). 

One  problem  with  syntactically-driven  approaches  to  structuring  knowledge 
acquisition  interactions  is  that  they  can  confuse  or  irritate  the  DE  because 
transitions  and  emphasis  may  not  cor. espond  to  his  Intuitions  about  the 
structure  of  the  domain  knowledge.  Until  the  KE  becomes  more  familiar  with 
the  domain  and  can  use  semantic  and  pragmatic  knowledge  to  guide  elicitation, 
his  options  for  overcoming  the  problems  of  syntactically-driven  interactions 
are  limited. 
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One  method  of  improving  dialogue  management  involves  varying  the  focus  of 
elicitation.  At  the  least,  the  KE  can  employ  some  content-free  heuristics  to 
modulate  repetitive  patterns  of  interaction,  which  can  easily  result  from 
strictly  depth-  or  breadth-first,  top-down  exposition  of  task  goals  and 
methods.  Depending  on  the  modularity  of  the  system's  intended  capabilities, 
the  KE  might  also  direct  elicitation  toward  knowledge  specific  to  each 
capability  in  turn,  perhaps  in  order  of  importance  or  of  planned 
implementation. 

Alternatively,  the  KE  can  use  syntactic  methods  in  a  bottom-up  manner 
that  gives  the  DE  more  initiative  in  controlling  transitions  and  emphasis. 

The  most  prevalent  method  is  to  orient  interactions  around  real  or 
hypothetical  task  situations  or  cases.  Using  case  generation,  even  a 
relatively  inarticulate  HE  can  usuaTTy  share  the  initiative  with  the  KE .  Once 
the  Of  has  described  an  "important"  case  and  how  it  should  be  treated,  the  KE 
can  use  syntactic  methods  to  extend  and  elicft  the  knowledge  necessary  to 
analyze  that  case.  When  appropriate,  the  KE  can  use  structural  features  of  a 
representation  to  prompt  for  related  cases.  For  example,  the  KE  might  ask, 

"Is  there  a  case  in  which  the  procedure  you  just  described  is  omitted?",  "Is 
there  a  case  in  which  those  two  goals  have  the  opposite  priority?",  or  "Is 
there  a  case  in  which  those  data  have  different  implications?".  This  type  of 
prompting  must  be  undertaken  judiciously,  since  such  questions  may  be 
misdirected  and  anomalous  from  the  DE's  perspective.  Until  he  acquires  some 
semantic  and  pragmatic  knowledge  about  the  domain,  the  KE’s  best  tactic  is  to 
promote  the  DE's  generation  of  interesting  different  cases.  The  KE  can  then 
build  his  knowledge  of  the  domain  by  inferring  the  dimensions  along  which  the 
">!'  chooses  to  elaborate  and  sequence  cases. 

There  a*-p  at  least  three  limitations  of  relying  on  interactions  built 
around  cases.  First,  multiple  cases  may  contain  much  redundant  knowledge, 
rath  subsequent  case  therefore  provides  less  new  information,  while  the  time 
required  to  discuss  them  may  not  decrease  appreciably.  It  thus  becomes  less 
efficient  over  time.  Second,  the  interaction  over  a  case  may  cover  many  more 
considerations  than  required  for  the  targeted  capabilities.  This  is,  of 
course,  a  function  of  how  narrow  the  target  set  is  in  relation  to  the  full 
task  and  of  how  much  context  is  required  to  present  a  case  meaningfully. 

Third,  the  focus  of  the  particular  cases  generated  by  the  DE  may  not  be 
sensitive  to  any  priorities  or  plans  the  KE  has  developed  for  working  on 
different  capabi 1 i ties . 

These  difficulties  imply  that  the  KE ' s  dialogue  management  plan  for 
knowledge  acquisition  can  emphasize  case-ori ented  elicitation  early  in  the 
process.  As  the  knowledge  base  grows  and  other  methods  for  effectively 
controlling  focus  become  available,  case  discussions  should  be  used  less 
frequently.  Skilled  KEs  generally  rely  on  a  combination  of  elicitation 
methods  including  (1)  dialogues  structured  by  applying  general  task 
description  frameworks  and  formal  representation  languages  and  (?)  bottom-up 
case -structured  dialogues  in  which  syntactic  structures  are  used  to  derive  and 
execute  local  plans  for  the  interaction.  However,  the  overall  set  of 
activities  cannot  be  planned  firmly.  Instead,  changes  in  interaction  mode  are 
triggered  by  the  KE ' s  perception  of  the  effectiveness  of  the  methods  in  use 
for  cueing  the  DE  and  the  rate  at  which  new  knowledge  is  being  elicited.  At 
present,  there  is  no  adequate  model  for  how  the  KF.  makes  these  determinations. 
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Another  aspect  of  the  Knowledge  Acquisition  process  worthy  of  note 
involves  feedback  opportuni ties  for  the  KE  and  DE.  When  the  process  is 
undertaken  as  a  series  of  protracted  interactions,  the  KE  may  accumulate  a 
body  of  knowledge  and  be  uncertain  as  to  its  completeness,  consistency,  and 
ability  to  support  the  targeted  capabilities.  He  cannot  provide  the  DE  with 
assurances  about  progress,  yet  such  feedback  is  desirable  for  maintaining  the 
DE's  motivation.  Coupling  Implementation  with  Knowledge  Acquisition  (and 
Formalization)  is  a  means  for  both  the  KE  and  DE  to  obtain  feedback  that  may 
alter  their  approach  to  Knowledge  Acquisition.  When  systems  are  built  in 
development  languages  that  facilitate  the  rapid  coding  and  testing  of 
prototypes,  the  ability  to  quickly  produce  and  review  initial  results 
typically  elicits  enthusiasm  and  increased  motivation  from  DEs.  The 
requirement  for  feedback  in  Knowledge  Acquisition  is  thus  another  reason 
knowledge  engineering  has  typically  been  iterative  and  incremental. 

To  summarize,  the  pursuit  of  Knowledge  Acquisition  is  closely  related  to 
the  pursuit  of  other  objectives.  Definition  determines  what  knowledge  needs 
to  be  acquired.  Formal i za tion  provides  structure  for  managing  knowledge 
acquisition  interactions  and  for  monitoring  completeness  and  consistency  of 
acquired  knowledge.  Implementation  provides  further  feedback  on  completeness 
and  consistency  and  enables  a  more  goal-oriented  pursuit  of  knowledge. 

C .  Forma  1 i z a t i on  Obj ec t i  ve 

Subobject i ve s 

Select  a  knowledge  representation  language 

Characterize  structure  of  acquired  knowledge 
Contrast  knowledge  characteristics  with  language  features 
Translate  informal  knowledge  descriptions  into  formal 

representation  language 

The  Formalization  objective  encompasses  the  activities  that  create  a 
formal  description  of  the  knowledge  acquired  from  the  DE  using  a 
representation  language.  Ideally,  the  KE  abstracts  certain  features  of  the 
acquired  knowledge  and  selects  (or  designs)  a  congruent  representation 
language.  However,  there  are  no  objective  criteria  for  determining  an 
appropriate  representation  formalism  for  knowledge  with  given  character! sti cs . 
Formalization  is  ad  hoc  with  respect  both  to  the  definition  of  knowledge 
charac teri st ics  and  WKat  they  imply  for  representation.  Each  KE  uses 
guidelines  based  on  his  knowledge  of  previous  knowledge  engineering  efforts 
and  of  knowledge  representation  research  in  general. 

While  representations  have  considerable  similarity  in  their  expressive 
power,  they  have  idiosyncratic  strengths  and  weaknesses.  The  weaknesses 
sometimes  prove  a  representation  unsuitable  for  easily  expressing  specific 
knowledge  or  for  supporting  satisfactory  implementation  of  a  system 
capability.  Since  there  is  no  general  capability  for  automatically 
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translating  knowledge  expressed  in  one  representation  to  another,^  choosing  a 
representation  that  later  demonstrates  weaknesses  can  create  problems  for  a 
knowledge  engineering  effort.  Only  very  recently  has  there  been  any  attempt 
to  propose  general  principles  for  matching  knowledge  characteristics  to 
representational  capabilities  (Buchanan,  et  al_. ,  1983;  Stefik,  et  al_.,  1983). 

The  principles  that  have  been  proposed  for  determining  a  representation 
language  are  derived  from  successes  and  failures  of  prior  efforts  rather  than 
from  theoretical  analysis.  Stefik  et  al .  (1983)  consider  three  broad  aspects 
of  the  acquired  knowledge:  the  hypotKesTs  space,  the  problem-solving  process, 
and  the  data.  These  correspond  to  the  goals,  procedures,  and  data  of  general 
task  description  frameworks.  The  important  dimensions  of  the  hypothesis  space 
are  its  size  and  structure,  where  structure  refers  to  dimensions,  such  as 
temporal ,  logical,  and  semantic,  upon  which  the  space  can  be  decomposed  and 
operated.  The  problem  solving  process  is  characterized  with  respect  to  its 
homogeneity  and  its  dominant  method  (e.g.,  backward-chaining,  forward¬ 
chaining).  Data  are  characteri zed  with  respect  to  their  certainty, 
completeness,  and  stability. 

The  guidelines  presented  by  Stefik  et  al .  (1983)  are  useful,  but  are  only 
a  first  approximation  to  an  understanding  oT~how  to  select  or  design 
representations  for  an  expert  system.  They  are  derived  from  only  limited 
experience  with  the  potential  set  of  problems  to  which  knowledge-based  systems 
might  be  applied.  Even  within  the  range  of  problems  that  have  been 
considered,  the  guidelines  are  not  unique  or  precise  prescriptions  and  cannot 
replace  the  knowledge  a  skilled  KE  uses  to  select  a  representation  for  use  in 
a  new  system. 

The  development  of  practical  knowledge  about  the  strengths  and  weaknesses 
of  different  knowledge  representations  for  different  types  of  problems  has 
been  impaired,  probably  more  than  any  other  applied  knowledge,  by  the  fact 
that  most  efforts  to  develop  expert  systems  have  been  dominated  by  the 
concerns  of  academic  research.  The  rewards  have  been  greater  there  for 
designing  new  representation  languages  and  associated  computational 
environments  than  for  intensively  analyzing  the  appl icability  of  existing 
ones.  Consequently,  there  has  been  a  proliferation  of  representation 
languages  and  systems,  few  of  which  have  been  applied  seriously  to  more  than 
one  or  two  problem  domains  or  have  been  used  outside  the  institution  in  which 
they  were  developed. 

There  is  also  a  practical  aspect  to  the  result  of  coupling  expert  systems 
efforts  to  inventive  research  on  knowledge  representation.  Few  representation 
languages  have  been  implemented  in  transportable,  production-quality 
computational  systems.  Thus,  unless  he  has  the  skill  and  resources  to  develop 
his  own  implementation,  a  practicing  KE's  choice  of  languages  is  limited.  If 


^Translating  between  representations  is  a  problem  similar  to  translating 
between  conventional  algebraic  prograwni ng  languages.  Despite  great  interest, 
little  progress  has  occurred  on  the  latter  problem  and  there  is  little  reason 
to  believe  that  the  capability  will  be  achieved  first  for  the  more  complex  Al 
representation  languages. 
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he  does  not,  then  a  priori  constraints  on  the  hardware  and  software 
environment  (e.g.,  as  imposed  by  the  customer)  can  further  limit  his  options. 
When  the  choice  of  representation  systems  is  known  to  be  limited,  these 
constraints  are  best  evaluated  during  Initial  efforts  on  Definition.  Doing  so 
can  allow  the  KE  to  use  knowledge  of  hardware  or  software  limitations  to 
circumscribe  the  system's  capabilities  and  the  types  of  knowledge  the  system 
will  require. 

Once  a  representation  language  has  been  selected  or  designed,  the  KE  can 
use  it  to  codify  informally  represented  knowledge  elicited  from  the  DE.  This 
is  not  necessarily  straightforward,  since  there  may  be  alternative  ways  of 
expressing  the  same  knowledge  within  a  representation.  A  common  uncertainty 
is  the  granularity  with  which  knowledge  is  to  be  described;  that  is,  what 
knowledge  will  be  considered  primitive  and  what  will  be  composite.  The  KE 
must  determine  granularity  such  that  the  representation  of  knowledge  is 
transparent.  Transparency  means  that  the  expression  of  the  knowledge  should 
not  be  excessively  decomposed  or  contorted  from  the  DE's  natural  way  of 
expressing  it.  Transparency  is  important  in  facilitating  debugging  and 
maintenance  of  a  system  and  for  making  the  system's  behavior  more 
understandable  to  its  users.  In  particular,  it  helps  make  explanation 
capabilities  in  the  system  congruent  with  users'  conceptions  of  an  appropriate 
level  of  abstraction  for  understanding  problems  in  the  task  domain. 

When  the  KE  has  selected  the  representation  language  and  has  begun  to 
encode  some  knowledge  into  it,  he  can  couple  more  closely  the  activities  of 
acquiring  and  encoding  knowledge,  bypassing  intermediate  informal 
descriptions.  As  described  above,  this  provides  mechanisms  for  dialogue 
management  using  the  syntax  of  the  representation.  The  KE  must  still, 
however,  interpret  and  integrate  the  knowledge  elicited  from  the  DE. 

0.  Design  Objective 

Subobjectives 

Develop  system  software  architecture 

Select  system  hardware  architecture 

The  Design  Objective  entails  formulating  the  hardware  and  software 
architecture  of  the  system.  The  process  of  producing  a  design  interacts  with 
and  depends  on  activities  in  the  Definition,  Formalization,  Implementation, 
and  Testing  Phases.  The  resulting  design  also  depends  on  compatibi 1 1 ty ,  cost, 
or  equipment  constraints  imposed  by  the  customer.  Constraints  on  hardware  and 
software  options  may  limit  Design  options  and  even  affect  the  capabilities 
that  can  be  incorporated  into  the  system.  In  the  absence  of  strong  customer 
constraints,  the  KT  can  design  a  software  and  hardware  architecture  based  on 
the  most  effective  match  between  target  capabilities  and  hardware/sof tware 
tools  and  techniques  to  realize  the  capabilities.  If  he  cannot  customize  a 
representation  language,  he  will  be  limited  to  archi tectures  that  can  be 
realized  in  an  existing  production-quality  language  system.  In  either  case, 
the  limited  number  of  prior  cases  from  which  the  KE  can  draw  can  create 
uncertainty  about  the  appl icability  of  any  particular  design  to  the  current 
problem  and  the  expected  performance  of  the  system  when  implemented.  Prior  to 
implementation,  the  type  and  scope  of  knowledge  obtained  during  Knowledge 
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Acquisition  will  suggest  to  the  KE  the  applicability  of  the  various  standard 
software  design  options  available  {e.g.,  type  of  inference  system,  structure 
of  the  knowledge  base,  representation  of  problem-sol ving  strategies, 
explanation  facilities)  and  hardware  options  available  (e.g.,  graphics  display 
vs.  character  display,  keyboard  input  vs.  mouse  input,  dedicated  processor  vs. 
time-shared  system). 

A  major  task  in  system  design  is  the  integration  of  the  representation 
language  system  that  encodes  the  task  expertise  with  satisfactory  user 
interface  software  and  hardware  to  provide  the  system's  interaction 
capabilities.  If  the  KE  is  constrained  to  use  a  particular  software  system, 
then  the  representati on  is  a  given.  Otherwise,  he  must  use  basic  computer 
science  skills  and  AI  knowledge  to  generate  a  design  that  will  be  both 
adequate  and  efficient.  While  existing  development  systems  from  which  the  KE 
may  select  his  programming  environment  offer  user  interface  facilities,  most 
are  tailored  to  the  needs  of  a  researcher  with  AI  programming  skills  rather 
than  a  customer  or  DE.  Hence,  in  developing  a  finished  system  for  a 
commercial  or  military  application,  the  KE  will  most  likely  face  a  significant 
design  effort  for  the  user  interface  features. 

Most  of  the  high-level  guidelines  and  knowledge  to  be  used  in  interface 
design  (e.g.,  partitioning  of  initiative)  should  have  been  garnered  during 
Definition  through  discussion  among  the  participants.  Lower-level  design 
considerations  (e.g.,  formats,  dialogue  management,  heuristics,  etc.)  may  be 
specified  during  initial  Definition,  but  more  likely  will  emerge  during 
Knowledge  Acquisition  and  Formalization.  These  phases  will  specify  in  detail 
the  type  and  format  of  knowledge  to  be  acquired  from  the  user  or  provided  by 
the  system  to  the  user,  as  well  as  the  structure  imposed  on  that  knowledge. 
This  specification  will  enable  the  design  of  interaction  protocols  from  which 
detailed  user  interface  features  and  characteristics  can  be  developed. 

However,  interface  design  for  knowledge-based  systems  (or  computer  systems  in 
general )  is  not  a  mature  discipline;  again,  the  knowledge  the  KE  can  bring  to 
bear  is  incomplete  and  heuristic.  In  most  cases,  testing  by  the  DE  and 
ini tial  use  by  the  intended  user  population  are  required  to  revise  the  design 
and  implementation  of  the  interface — another  reason  for  the  reliance  on 
incremental  system  development.  In  the  expert  systems  field  it  is  typical  to 
defer  low-level  interface  design  and  implementation  until  the  implementation 
of  the  "Fundamental  problem-sol ving  capabilities  is  relatively  robust  and  ready 
for  use  outside  the  development  setting. 

E_.  Implementation  Objective 

Subobjectives 

Implement  inferential  capabilities  and  user  interface 

Encode  domain  knowledge 

Detect  and  debug  programming  errors 

Implementation  entails  coding  ti>e  formalized  knowledge  acquired  from  the 
'E  in  the  selected  hardware- software  environment  and  verifying  that  the  system 
runs  without  software  errors.  (Such  testing  is  distinguished  from  that 
undertaken  during  the  Testing  phase  which  assesses  the  system's  effectiveness 
in  realizing  the  target  capabilities.)  The  Implementation  process  provides 
the  KE  with  information  that  can  require  additional  work  and  revision  in  the 
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Definition,  Knowledge  Acquisition,  Formalization,  and  Design  phases.  This 
information  bears  upon  decisions  that  prior  to  implementation  were  based  on 
incomplete  and  uncertain  knowledge  generalized  from  prior  system  development 
efforts.  The  utility  of  such  information  is  a  major  reason  that 
Implementation  has  been  undertaken  incrementally,  concurrent  with  the  pursuit 
of  other  objectives,  in  many  efforts  to  build  knowledge-based  systems.  A 
general  principle  for  knowledge  engineering  urged  upon  would-be  KE's 
(Buchanan,  et  aK,  1983)  is  the  early  implementation  of  an  experimental 
prototype. 

The  extent  to  which  the  KE  must  code  the  basic  software  environment,  as 
opposed  to  just  the  formalized  domain  knowledge,  depends  on  the  representation 
language  he  has  selected.  User  interface  software  must  generally  be  developed 
even  if  the  KE  can  adopt  an  existing  language  system,  but  its  design  and 
implementation  are  usually  deferred  until  the  system’s  problem-solving 
capabilities  have  been  implemented  and  tested.  Thus,  at  worst,  if  he  is 
implementing  his  own  basic  software  environment,  the  KE  must  only  implement 
the  representation  language,  the  associated  problem-solving  mechanisms,  and  a 
skeletal  interface  before  starting  to  encode  the  formalized  domain  knowledge. 
However,  experience  indicates  that  including  specialized  editors  and  debugging 
tools  in  the  software  environment  greatly  enhances  the  efficiency  of 
incremental  implementation  efforts. 

Of  course,  the  KE’s  approach  to  implementation  also  depends  upon  the 
availability  of  the  OE.  Ready  access  to  the  DE  supports  an  incremental 
approach,  especially  where  an  early  selection  of  representation  language  can 
be  made.  Limited  access--intensi ve,  infrequent  meetings  spaced  over  time-- 
reduces  the  opportunity  for  incremental  implementation  and  thus,  given  the 
current  state-of-the-art  in  knowledge  engineering,  the  effectiveness  of  the 
system-building  effort.  Except  in  cases  where  the  KE  has  been  able  to  adopt  a 
complete  existing  architecture  for  a  new  problem  domain,  successful  efforts 
have  included  a  OE  as  a  regular  member  of  the  system-building  team  for  the 
full  duration  of  the  project  effort. 

Incremental  implementation  and  the  concurrent  pursuit  of  other  objectives 
is  inevitable,  even  desirable,  as  long  as  the  KE’s  knowledge  is  as 
unsystematic,  incomplete,  and  uncertain  as  it  is  at  present.  The  incremental 
approach  provides  the  KE  with  rapid  feedback  that  can  provide  early 
indications  of  the  advisability  of  revising  system  design  decisions.  The 
partial  working  system  also  serves  an  organizational  function  in  helping  to 
maintain  and  promote  the  motivation  of  the  system  building  team. 

E.  Testing  Objective 

'.ubohjectfves 

formulate  diagnostic  test  cases 

Assess  system  performance  on  test  cases 

Collect  data  on  user's  interactions  with  the  system 

Determine  system  performance  on  other  targeted  capabilities 

Provide  information  required  for  effective  revisions 

Once  a  meaningful  subset  of  the  formalized  domain  knowledge  is 
implemented,  the  KE  can  pursue  Testing  of  its  targeted  problem-solving 
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capabilities.  The  typical  method  of  testing  is  for  the  KF.  and  DE  to  define 
critical  (i.e.,  topical,  important,  frequent,  or  difficult)  problem  situations 
for  which  the  OF  can  specify  criteria!  performance.  Normally,  testing  with 
these  cases  is  repeated  at  points  during  the  growth  of  the  knowledge  base  and 
whenever  the  architecture  or  implementation  of  problem-solving  capabilities  is 
otherwise  modified,  as  a  check  for  new  conflicts  and  inconsistencies  that  may 
have  been  introduced. 

A  second  aspect  of  Testing  involves  evaluation  of  user  interface 
capabi 1 i ties .  While  these  can  be  tuned  initially  by  the  development  team,  use 
by  others  from  the  intended  user  community  is  required.  This  testing  has 
typically  been  limited  and  informal  in  many  prior  efforts  conducted  in 
research  environments. 

The  results  of  Testing  can  bear  upon  decisions  and  knowledge  accumulated 
in  pursuit,  of  each  of  the  other  objectives  of  the  system-building  effort. 

They  may  lead  the  KE  to  determine  that: 


a.  Particular  capabilities  are  inappropriate  or  intractable  and 
should  be  eliminated,  perhaps  to  be  replaced  by  others. 

b.  The  domain  knowledge  is  incomplete,  inconsistent,  or  inaccurate. 

c.  The  representation  language  lacks  the  required  expressive  power 
or  transparency. 

d.  The  hardware-software  architecture  lacks  sufficient  capacity  to 
•■un  the  system  or  speed  to  execute  it  at  an  acceptable  rate. 

The  problem-solving  methods  and  strategies  are  inadequate--that 
is,  the  system's  behavior  is  not  "expert." 

H  nsat  i  '..factory  performance  on  criteria!  cases  might  be  a  symptom  of  any 
of  these  problems.  Initially,  the  KF.  responds  by  iterating  on  Knowledge 
Acquisition  and  Implementation,  since  these  are  most  accessible  to  revision. 

vent.ijal  ly ,  however,  if  problems  persist  or  if  new  problems  continue  to 
'merge,  the  KF  need  to  consider  modification  of  the  Definition,  Formal icat ion , 
<i r  Design  objectives.  Such  modifications  may  have  more  far-reaching 
ramifications  for  the  nature  of  the  resulting  system.  When  system  developers 
opt  for  new  capabilities,  representation  languages,  or  architecture,  the 
domain  knowledge  base  usually  requires  fundamental  changes  that  entail  total 
redesign  and  reimplementation  of  the  initial  prototype  (Buchanan,  et  aK , 
1081). 


this  process  of  test  and  revision  is  yet  another  reason  for  the 
desirability  of  incremental  design,  implementation,  and  test.  By  testing 
performance  as  the  system  evolves,  it  may  bo  possible  to  accelerate  the  point 
where  reimplementation,  if  It  is  requ'red,  can  occur  prior  to  a  large 
investment  of  time  and  resources  on  the  initial  prototype  development  effort. 
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SECTION  III 

CURRENT  TECHNOLOGIES  FOR  ASSISTING  KNOWLEDGE  ENGINEERING 


The  model  of  the  knowledge  engineering  process  presented  in  the  previous 
section  provides  a  framework  for  consideration  of  alternative  approaches  to 
automating  portions  of  the  KE's  tasks.  Any  new  proposals  and  designs  for  such 
automated  systems  must  consider  previous  and  current  attempts  to  provide 
assistance  to  DEs  and  KEs  during  system  development.  This  section  selectively 
reviews  this  body  of  research. 

Two  general  approaches  have  been  adopted  to  facilitate  the  knowledge 
engineering  process  (Barr  and  Feigenbaum,  1982).  The  first  seeks  to 
facilitate  the  interactive  transfer  of  expertise  (ITE),  those  phases  of  the 
knowledge  engineering  effort  during  which  the  KF  elicits,  formalizes,  and 
encodes  the  DL ' s  relevant  domain  knowledge.  Some  systems  following  this 
approach  aim  to  assist  the  dialogue  management,  bookkeeping,  and  translation 
performed  by  the  K[  .  These  systems  "interview"  the  user  to  collect  domain 
knowledge.  Other  systems  provide  high-level  programming  languages  specially 
designed  to  capture  and  represent  heuristic,  rule-based  expert  knowledge.  In 
other  cases,  attempts  have  been  made  to  develop  knowledge  base  checking 
facilities  for  evaluating  the  consistency  and  completeness  of  the  encoded 
knowledge.  Finally,  some  work  has  been  directed  toward  supporting  the 
definition  of  system  capabilities  and  the  formulation  of  a  software  design. 
Development  of  such  aids  in  many  cases  has  evolved  in  the  context  of  a  larger 
project  to  implement  a  particular  expert  system;  hence,  such  projects  have 
often  had  a  pragmatic,  ad  hoc  nature. 

The  second  approach  to  facilitating  development  of  expert  systems  is 
"automatic  theory  formation"  or,  simply,  automated  learning.  That  research 
aims  to  develop  systems  that  induce  new  knowledge  from  experience--for 
example,  hy  analyzing  the  behavior  of  the  DE  on  particular  test  cases.  (The 
Meta-Dcndral  System  [Lindsay,  et  al.,  1900]  illustrates  this  second  approach). 
These  systems  would  replace  the  ITF  process  as  a  method  for  developing 
knowledge  bases  and  would  apply  to  problems  in  which  ITF  is  not  productive, 
either  because  there  is  no  consensus  among  DEs  or  because  the  DE  cannot 
articulate  his  knowledge.  Work  on  automated  theory  formation  has  typically 
been  more  theoretical  and  to  a  large  extent  has  been  pursued  independently 
from  practical  applications  of  expert  systems.  Because  its  applied  (as 
opposed  to  theoretical )  focus  matches  the  concerns  of  the  present  project,  our 
review  considers  only  those  efforts  directed  at  ITE. 

The  developers  of  the  first  expert  systems  quickly  faced  the  problems  of 
managing  the  growth  of  their  system  and  of  providing  facilities  for  end  users 
to  control  and  maintain  them.  KEs  found  that  once  a  representation  language 
and  system  architecture  had  been  selected,  the  task  of  eliciting  and  encoding 
new  knowledge  was  more  systematic,  albeit  still  difficult,  using  generic  tools 
provided  within  AI  programming  systems.  They  responded  by  developing 
special izations  of  these  tool s--knowledge  structure  editors,  break  and  trace 
packages  for  debugging,  file  managers— geared  to  the  representations  and 
capabilities  of  their  expert  system.  In  so  doing,  they  endeavored  to  provide 
the  means  for  altering  the  expert  system  using  the  concepts  in  the  application 
domain  instead  of  the  programming  primitives  that  implemented  that 
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appl i cation. 

Following  the  early  research  on  ITE  conducted  in  specific  application 
domains,  a  second  generation  of  efforts  has  attempted  to  extend  the  initial 
work  and  develop  generic  tools  for  constructing  expert  systems.  Such  tools 
have  taken  the  form  of  either  skeletal  systems  or  specialized  programing 
languages.  The  skeletal  systems  provide  a  fixed  problem-solving  architecture 
(knowledge  base  representati ons  and  control  mechanisms)  and  user-support 
modules  for  building  knowledge  bases  using  the  system.  Use  of  such  skeletal 
systems  is  limited  to  problem  domains  for  which  their  capabilities  are 
congruent  with  the  customer's  needs  and  their  representations  adequate  to 
capture  the  structure  and  content  of  the  domain  knowledge.  The  specialized 
programming  languages  provide  data  and  control  constructs  better  tailored  to 
‘.he  knowledge  structures  used  in  expert  systems  than  the  constructs  of  general 
M  languages  such  as  LISP.  These  languages  are  more  general  than  skeletal 
systems,  enabling  more  flexible  selection  of  archi tectures  and  capabilities. 
However,  they  require  the  user  to  design  representati ons  and  an  overall 
architecture  and  consequently  to  engage  in  greater  implementation  effort. 

In  general,  skeletal  systems  aid  the  KE  in  achieving  the  objectives  of 
Knowledge  Acquisition,  Implementation,  and  Testing,  while  they  eliminate 
Design  effort.  Specialized  programming  languages  support  Implementation 
directly  and  Design  and  Testing  indirectly  through  the  transparent  high-level 
constructs  they  provide  to  the  KE . 

All  the  efforts  to  aid  I TF  have  been  oriented  toward  either  KEs  or  DEs 
with  considerable  sophistication  about  the  expert  system  they  are  helping  to 
develop  or  maintain.  Use  of  ITE  aids  by  DEs  with  no  KE  support  has  been 
achieved  only  for  relatively  mature  systems  in  which  the  user's  role  is  to 
update  and  extend  an  already  extensive  knowledge  base.  The  use  of  skeletal 
systems  for  building  a  new  knowledge  base  and  the  use  of  specialized 
programming  languages  remains  limited  to  KEs.  However,  the  availability  of 
such  systems  and  languages  reduces  the  level  of  programming  ability  a  KL  needs 
to  implement  an  expert  system. 

The  following  discussion  of  representati v«  systems  is  divided  into  two 
parts.  The  first  describes  systems  in  which  support  for  ITr  was  developed  for 
a  particular  architecture  arid  problem  domain.  The  second  describes  mor» 
general  efforts  embodied  in  skeletal  systems,  specialized  programing 
languages,  and  other  general  systems  for  impl ementi ng  task  expertise. 

APPLICATION-SPECIFIC  ASSISTANCE  FOR  ITE 

7EIRISIAS.  TEIRFSIAS  (Davis,  1977)  was  developed  as  a  subsystem  to  support, 
the  growth  and  maintenance  of  the  MYCIN  system  { Short! i ffe ,  1976)  knowledge 
base.  Its  facilities  include  tools  for  modification  and  debugging  of  the 
production  rules  MYCIN  uses  to  encode  knowledge  about  the  relationships 
between  clinical  data,  infectious  diseases,  and  drug  therapies. 

!  F I  ID  S  I  A!,  ’  mode  of  interaction  is  a  mixed-initiative  dialogue  using  i 
mixture  of  gu.is i -F rigl i sh  language  and  menu  interactions  with  the  user.  The 
dialogue  model  casts  the  user  in  the  role  of  a  teacher  who  is  instructing  the 
system  about  new  domain  rules.  The  system  assumes  the  role  of  a  motivated 
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learner,  raising  questions  of  clarification  about  what  it  is  told  based  on  the 
relationship  of  the  new  rules  to  old  ones  it  knows.  For  example,  it  will 
point  out  to  the  user  that  a  new  rule  does  not  contain  a  predicate  or  action 
clause  that  occurs  in  other  rules  referencing  the  same  concepts. 

TE I  RE  S I AS '  mechanisms  depend  on  knowledge  it  has  about  concept  categories 
and  rule  structure.  This  knowledge  allows  it  to  parse  quasi -Engl i sh  rule 
descriptions  into  its  internal  representation.  It  also  permits  TE I RE S IAS  to 
raise  questions  about  the  semantics  and  pragmatics  of  proposed  rule 
modi fications  or  additions.  This  application-specific  meta-knowledge  was 
initially  programmed  into  TEIRESIAS,  but  it  has  mechanisms  for  modifying  the 
knowledge.  For  example,  knowledge  of  concepts  frequently  referenced  together 
is  induced  dynamically  as  the  knowledge  base  changes. 

To  support  debugging,  TEIRESIAS  interfaces  to  MYCIN's  explanation 
facilities  and  case  library.  When  modifying  or  adding  rules,  the  user  can  ask 
for  a  dynamic  trace  of  rule  tests  to  determine  why  specific  conclusions  were 
or  weren't  reached.  This  aids  the  user  in  revising  existing  rules  when  new 
rules  are  added,  so  that  undesirable  interactions  among  rules  can  be  corrected 
as  the  knowledge  base  grows. 

TEIRESIAS'  approach  is  based  on  two  fundamental  assumptions.  The  first 
assumption  is  that  the  host  system's  (i.e.,  MYCIN's)  control  structure  and 
knowledge  representations  are  understandabl e  abstractions  of  the  domain  for 
its  users.  TEIRESIAS'  users  were  the  same  KEs  and  DEs  who  had  built  MYCIN 
originally.  They  were  therefore  familiar  with  MYCIN's  abstractions  of  the 
problem  domain  and  problem-solving  process  and  could  think  in  terms  of  them, 
even  if  they  were  not  completely  intuitive.  TEIRESIAS  would  not  be  accessible 
to  use  by  others  without  considerable  training  about  how  domain  knowledge  is 
represented  and  used  by  MYCIN. 

TEIRESIAS’  use  of  meta-knowledge  about  the  host  system's  implementation 
reflects  a  second  assumption:  that  MYCIN's  control  structure  and  knowledge 
representations  were  a  sound  approach  and  therefore  stable.  The  architecture 
and  interface  of  the  ITE  functions  are  directly  dependent  on  those  of  the  host 
system  and  would  require  revision  if  the  latter  were  modified.  MYCIN  was  in 
fact  a  relatively  mature  system,  with  a  significant  existing  knowledge  base, 
at  the  time  TEIRESIAS  was  designed  and  implemented.  The  value  of  implementing 
ITE  support  for  a  mature  system  depends  on  whether  the  knowledge  in  the 
application  domain  is  expected  to  change  over  time,  on  how  complex  the 
"manual"  process  of  modifying  the  knowledge  base  is,  and  on  the  skills  of  the 
individuals  who  are  available  to  implement  modifications. 

FAS.  The  ‘‘nowledge  Acquisition  System  (KAS)  (Reboh,  1981)  was  designed  as  an 
Ht  aid  for  the  PROSPECTOR  system  (Duda,  et.  al.,  1978).  like  TEIRESIAS,  KAS 
wa-.  designed  and  implemented  only  after  (Ts  host  system's  architecture  was 
■  fable  and  a  subs t.ant  i  \  1  knowledge  base  existed.  Similarly,  its  architecture 
and  interface  are  dependent  on  the  representations  and  control  structures  of 
PROSPI.  i.  TOP. .  However,  it  does  not  us..  application  dependent  meta-knowledge 
about  the  PROSPECTOR  problem  domain  (i.e.,  i nterpreta ti on  of  geological  data). 
Thus,  KAS  is  transportable  with  the  bare  PROSPECTOR  architecture  (i.e.,  as  a 
skeletal  system)  to  other  appl ications. 
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KAS  takes  advantage  of  PROSPECTOR'S  network  knowledge  representation  for 
guiding  the  acquisition  of  related  rules.  In  particular,  it  supports  the 
joint  efforts  of  a  KE  and  DE  to  define  a  new  i nference  model ,  which  describes 
the  rules  relating  geological  data  to  a  specifTc  type  of  ore  deposit. 
PROSPECTOR  uses  networks  to  partition  and  structure  its  rules  into  inference 
models,  and  KAS  uses  the  structural  syntax  of  the  network  representation  to 
stimulate  and  monitor  the  elicitation  of  models.  In  addition  to  supporting 
knowledge  acquisition  in  this  manner,  KAS  provides  the  KE  with  capabilities 
for  debugging  and  evaluating  the  performance  of  models  by  tracing  their 
execution  on  stored  cases  and  new  ones  provided  by  the  DE . 

KAS's  basic  approach  to  model  building  is  initial  top-down  definition  of 
the  abstractions  in  an  inference  model  follower  by  iterative  top-down 
elaboration  of  the  arcs  and  nodes  of  the  network  created  by  the  original 
definition.  These  elaborations  mc'ude  the  logical  semantics  of  the  arcs 
linking  nodes,  the  numeric  values  of  parameters  to  be  tested  against  data  and 
to  be  used  to  assign  confidence  to  inferences,  and  specifications  for  the 
format  of  questions  and  answers  PROSPECTOR  should  use  when  asking  its  user 
about  parameters. 

The  main  subsystem  of  KAS  used  in  model  building  is  the  REsident  Network 
Editor  (RENE),  which  incorporates  knowledge  about  PROSPECTOR'S 
representational  and  computational  formalisms.  RENE  provides  an  interactive 
editor  for  networks,  automatic  bookkeeping  for  the  model  implementation 
process,  dialogue  management  during  elicitation,  and  an  interface  for 
controlled  execution  of  inference  networks. 

The  structure  editor,  like  those  implemented  for  AI  programming 
languages,  provides  the  user  with  primitives  for  manipulating  and  altering 
complex  data  structures--in  this  case,  networks.  Commands  for  changing  the 
editor's  attention  and  for  modifying  knowledge  reference  arcs  and  nodes  '-athe 
‘ban  the  more  primitive  data  types  (e.g. ,  strings  of  characters,  lists,  atoms 
used  in  PROSPECTOR  to  implement  these  abstractions.  The  goal  is  hoth  to 
•'•-event  syntactic  errors  that  could  occur  if  the  user  manipulated  low- love' 
representations  directly  and  to  allow  the  user  to  manipulate  constructs 
corrospondi ng  to  those  he  uses  to  articulate  his  expertise  during  problem 
,ol  vi  ng . 

The  bookkeeper  is  an  "intelligent  agent"  that  "looks  over  the  shoulder  o 
the  KE"  as  he  interacts  with  RFNE's  other  facilities.  It  prevents  syntactic 
•‘trors--for  example,  by  noticing  that  the  KE  hasn't  connected  parts  of  an 
inference  network.  It  fills  in  defaults  for  descriptive  and  quantitative 
attributes  of  arcs  and  nodes  that  the  KE  does  not  supply.  It  alerts  the  KE  t 
’-ami  fications  of  inputs  for  prior  inputs.  The  bookkeeper  uses  a  hierarchy  of 
knowledge  about  representations,  about  the  host  system,  and,  to  the  extent 
that  it  has  been  supplied,  about  the  domain.  This  knowledge  is  used  by  a 
lata-cent.ered  programming  model  that  allows  a  set  of  actions  to  be  associated 
with  each  type  of  element  of  a  representation .  These  actions  (e.g., 
generating  a  default  value)  are  executed  whenever  an  element  is  added  or 
modified.  In  this  way,  RENE  generates  side-effects  for  actions  initiated  by 
the  user  that  reflect  the  system's  built-in  knowledge  of  possible  entailmerfs 
of  those  actions. 


KAS  primarily  employs  a  questi on-and-answer  dialogue  protocol,  generatin 
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its  output  from  pro-stored  templates.  It  uses  menus  and  a  quasi -Engl  i  sh 
command  1angu.nu’  for  user  inputs.  To  inhibit  user  errors  it  uses  the  grammar 
of  the  command  language  to  prompt  the  user  rather  than  attempting  to  parse 
full  statements.  The  grammar  interfaces  to  PROSPECTOR ' s  taxonomic  networks 
(networks  that  represent  the  concepts  used  in  the  inference  networks)  to 
extend  the  lexical  richness  of  the  command  language  as  the  knowledge  base 
grows . 

During  model  development,  RENT'S  capabilities  for  controlled  execution  of 
the  evolving  model  within  PROSPECTOR  allow  immediate  incremental  testing  and 
refinement.  Besides  helping  to  prevent  the  propagation  of  dependent  errors, 
the  independent  debugging  of  parts  of  a  model  provides  motivating  feedback  to 
the  KE  and  DE. 


Once  the  user  has  developed  an  initial  model,  KA$  can  he  used  to  install 
and  tune  it  within  the  existing  PROSPECTOR  knowledge  base.  Tuning  includes 
the  addition  of  mod" 7 -specific  elaborations  that  modify  defaults  supplied  by 
RENT,  force  specific  question  sequences,  and  control  the  focus  of  attention 
during  a  consultation  that  accesses  the  model.  At  this  stage,  KAS  also  offers 
facilities  f r-  “batch  node"  testing  of  the  complete  model  on  cases  supplied  by 
’he  expert.  These  fji  >  1  ities  ,»rovj<J«  automatic  variation  of  answers  stored 
f  j:'  the  cases  to  allow  a  sensitivity  analysis.  They  also  allow  the  mixing  of 
stored  answers  with  those  acquired  interactively  so  that  the  KE  and  DE  can 
more  easily  explore  a  particular  issue  in  variety  of  contexts. 

■“he  developers  of  k.-v;  have  drawn  several  general  conclusions  from  their 
effort.  r  > r$t,  regarding  the  knowledge  that  must  be  acquired,  support  for  I  IE 
should  extend  beyond  the  acquisition  of  domain  problem-solving  knowledge. 
Effective  interfaces  for  knowledge-based  systems  like  PROSPECTOR  require 
domain-speci fir  meta-knowledge  to  provide  dialogue  control  if  consultations 
are  to  follow  a  course  that  is  acceptable  to  its  users.  KA$  is  a  major  aid 
for  adding  and  testing  such  meta-knowledge  for  new  inference  models.  Second, 
use  of  an  existing  knowledge  base  has  important  benefits  fo--  supporting  the 
acquisition  of  new  knowledge.  In  particular,  experience  with  KAS  has  shown 
that  expanding  PROSPECTOR'S  network  of  taxonomies  without  regard  to  its  actual 
use  by  existing  inference  models  is  an  aid  to  acquiring  new  models.  This 
benefit  derives  from  the  use  of  the  taxonomies  to  dynamically  expand  the 
vocabulary  of  the  .ommand  language.  The  knowledge  a  system  already  contains 
is  also  important  for  supporting  the  understanding  of  incomplete  references  by 
the  user.  Without  this  capability,  unless  the  system  can  somehow  provide 
enough  <  no  text  for  the  user  to  keep  track  of  the  contents  of  the  knowledge 
base,  it  is  difficult  for  the  user  to  interact  with  a  knowledge  acquisition 
aid. 


rhe  final  conclusion  concerns  the  generality  of  knowledge  acquisition 
aids.  Effective  aids  require  knowledge  about  the  representations  used  in 
their  host  systems.  The  details  of  such  representations  depend  on  the  problem 
domain  and  purpose  encompassed  by  the  host  system.  bus,  the  generality  of 
specific  aids  can  never  be  greater  tnan  the  generality  of  their  host  system. 
PROSPECTOR/K AS  attempts  to  maximize  their  generality  through  a  "layered" 
architecture  that  localizes  the  modifications  needed  to  adapt  the  system  to 
different,  domains  and  purposes.  Several  attempts  to  adapt,  the  architecture  to 
different  classification  and  interpretation  problems  have  confirmed  that 
minimal  changes  are  required  to  support,  applications  of  PROSPECTOR/KAS  to 
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other  members  of  the  class  of  data  interpretation  problems. 

THE  ONCOCIM  KNOWLEDGE  BASE  VERIFIER.  ONCOCIN  is  an  expert  system  that 
provides  its  users  with  reconuiendations  about  cancer  treatment  therapies  for 
individual  cases.  Part  of  the  ONCOCIN  system  (Suwa,  Scott,  and  Shortliffe, 
1982)  is  an  ITE  aid  for  the  extension  and  revision  of  the  ONCOCIN  knowledge 
base.  This  aid  checks  the  completeness  and  consistency  of  the  knowledge  base 
whenever  it  is  modified.  Unlike  the  approach  taken  in  TEIRESIAS  and  KAS,  the 
ONCOCIN  knowledge  base  verifier  allows  testing  for  knowledge  consistency  and 
completeness  prior  to  the  existence  of  or  access  to  a  functioning  host  system. 

ONCOCIM  rules  determine  a  therapy  protocol  represented  as  a  list  of 
"action  parameter  values".  Each  rule  includes  a  context  and  a  set  of 
conditions  for  determining  its  applicability  to  a  case.  The  verifier  operates 
by  syntactically  analyzing  rules  that  recomend  the  same  action  in  the  same 
context  and  generating  a  table  whose  row  entries  are  all  the  possible 
combinations  of  condition  parameters  found  in  the  rules.  The  table  indicates: 


a.  Missing  rules:  combinations  of  condition  parameters  not 
associated  with  any  therapy  protocol  . 

b.  Conflict:  combinations  of  condition  parameters  that  would 
succeed  in"  “the  same  situation  and  are  associated  with  different 
protocols. 

c.  Redundancy  and  Subsumption:  combinations  of  condition  parameters 
that  wouTcT  YucceeJ  in  tTie~  same  situation  and  are  associated  with 
identical  protocols. 

Only  conflicts  are  true  errors,  but  the  other  data  serve  as  additional  foci 
for  a  review  and  revision  of  the  knowledge  base  by  the  KE  and  DE.  The  table 
display  constitutes  the  entire  user  interface  of  the  knowledge  base  verifier. 
Tnis  limited  interface  restricts  the  verifier's  utility  to  a  KE  who  has  a  dec 
understanding  of  its  operation. 

The  verifier  was  initially  developed  and  used  to  support  the  construction 
of  the  original  ONCOCIN  knowledge  base.  It  is  specific  to  ONCOCIN’s  rule 
representation,  but  independent  of  the  domain  semantics  and  system 
architecture.  The  generality  of  the  approach  does  seem  limited  by  domain 
characteristics,  however.  To  avoid  a  combinatorial  explosion  in  the  size  of 
the  table,  the  KE  must  partition  rules  by  "context"  into  sets  sharing  a  small 
number  of  condition  parameters.  For  the  discovery  of  missing  rules  to  be  a 
useful  activity,  most  combinations  of  condition  parameters  must  be  meaningful 
in  the  problem  domain. 


SFCS.  SECS  (Wipke,  et  al.,  1977)  is  an  expert  system  that  helps  its  users 
solve  organic  chemistry  synthesis  problems.  Its  knowledge  base  is  a  body  of 
rules,  called  transforms,  each  of  which  relates  a  target  substructure  to  its 
precursors  in  a  -eactfon.  SECS  includes  an  ITE  support  facility  that  allows 
its  DF  users  to  maintain  and  extend  its  knowledge  base. 

Modifications  and  additions  to  the  SFCS  knowledge  base  are  made  using  the 
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ALCHEM  language,  a  quasi-English  command  language  with  a  rigid  syntax.  The 
ALCHEM  commands  enable  the  description  and  manipulation  of  transforms  in  terms 
of  schema-like  abstractions.  Each  ‘'slot"  of  the  schema  has  its  own 
specialized  syntax  which  is  recognized  by  ALCHEM.  Although  the  ALCHEM  syntax 
is  rigid,  the  ITE  interface  is  flexible  in  that  the  user  has  full  control  over 
how  to  proceed  in  defining  a  new  transform  schema. 

SECS  and  ALCHEM  represent  an  extreme  approach  for  using  domain  and 
system-dependent  knowledge  in  implementing  support  for  ITE.  At  the  cost  of 
generality  and  of  considerable  conmitment  by  its  users  to  master  new  skills, 
the  result  has  been  a  mature  system  which  is  now  completely  maintained  by  its 
user  community. 

KOL .  The  Knowledgeable  Opponent  Librarian  ( KOL  )  ( A1 perovi tch ,  198?)  is  an  ITE 
aid  developed  for  use  with  a  knowledge-based  tactical  gaming  system.  It 
supports  the  elicitation  of  tactical  plans  that  can  be  executed  by  an 
automated  opponent  in  the  host  system. 

KOL's  elicitation  method  is  based  on  top-down  refinement.  The  user, 
either  a  KE  or  trained  QE,  is  prompted  to  describe  a  tactical  plan  at 
progressively  lower  levels  of  abstraction.  Each  refinement  expands  an  action 
into  a  number  of  more  specific  actions.  Ultimately,  the  refinement  must  reach 
a  level  where  the  descriptions  reference  only  a  predefined  set  of  domain- 
specific  primitive  elements. 

KOL  allows  the  user  to  exercise  initiative  and  return  to  previously 
expanded  nodes  in  the  plan  tree  in  order  to  enter  modi fications.  It  has 
bookkeeping  facilities  for  informing  the  user  when  such  modifications  have 
ramifications  for  other  nodes  and  for  monitoring  whether  all  branches  of  the 
plan  have  been  expanded  to  the  level  of  domain  primitives. 

The  KOL  user  interface  uses  stored  text  templates  to  generate  prompts. 

It  has  a  structure  editor  for  input  that  limits  syntactic  errors.  It  makes 
use  of  CRT  display  features  to  present  context  information  during  editing  and 
to  display  plans.  Those  features  are  also  used  during  elicitation  to  provide 
limited  context  around  the  plan  node  being  expanded.  The  interface  does  not 
provide  capabilities  for  usfng  existing  plans  to  build  new  ones.  In  addition, 
because  it  can  only  interpret  the  design  descriptions  at  the  primitive  level 
of  the  plan  tree  (i.e.,  the  leaves  of  the  tree),  it  can  provide  no  assistance 
in  evaluating  the  overall  design. 

KOL  is  essentially  an  aid  for  program  design  and  automatic  programming  in 
a  partic  lar  domain--that  of  submarine  tactics.  It  forces  the  user  to  design 
a  plan  (program)  specification  in  a  structured,  top-down  fashion.  It  then 
uses  an  interpreter  to  translate  the  lowest  level  of  the  design,  specified  in 
a  predefined  abstract  langua^,  into  Pascal  code  that  can  be  executed  by  the 
host  system. 

The  KOL  approach  to  supporting  ITE,  like  most  of  the  others  described 
thus  far,  is  viable  only  after  its  host  system  is  relatively  mature.  It 
depends  on  the  existence  of  a  stable  domain-dependent  set  of  primitive 
concepts  and  a  language  for  referencing  them.  The  KOL  implementation  is 
relatively  modular,  allowing  the  elicitation  and  bookkeeping  facilities  to  be 
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used  with  different  domain-specific  primitive  plan  languages. 

KOL's  generality  is  limited  to  problem  domains  in  which  plans  are 
conceived  as  hierarchical  and  in  which  the  actions  at  each  level  of  the 
hierarchy  are  sequential  and  independent.  This  approach  to  planning  may  be 
overly  restrictive  in  certain  situations  and  not  well  matched  to  humans' 
planning  style.  For  example,  for  some  problems  the  plans  may  not  be  most 
naturally  represented  as  hieiarchi cal  structures.  In  cases  in  which  the  plans 
are  hierarchical ,  actions  in  the  plan  may  include  concurrent  and  asynchronous 
actions.  In  addition,  planners  do  not  always  icv'lop  their  plans  in  strictly 
top-down  fashion.  Rather,  they  may  generate  plan  elements  opportunistically 
as  decisions  and  choices  at  different  levels  of  abstraction  occur  to  them. 
Therefore,  an  aid  for  ITT  should  provide  flexibility  and  robustness  in  the 
representation  structure  of  the  knowledge  to  be  elicited  and  in  the  degree  of 
user  control  over  what  portion  of  the  structure  to  elaborate  next.  The  need 
for  such  flexibility  increases  as  the  intended  generality  and  intended  scope 
of  applicability  of  the  aid  for  ITT  increases. 

! TF  SUPPORT  IN  APPLICATION-INDEPENDENT  KNOWLEDGE -ENGINEERING  TOOLS 

EMYCIN.  EMYCIN  (Essential  MYCIN)  (van  Melle,  1979)  is  a  skeletal  system 
derived  from  the  architecture  of  the  MYCIN  system.  It  has  been  applied  to 
classification  and  interpretation  problems  in  several  domains. 

EMYCIN  contains  all  the  support  facilities  for  ITT  that  TEIRESIAS 
provided  for  MYCIN,  except  those  that  were  directly  dependent  on  domain 
'•knowledge  programmed  into  TEIRESIAS  or  available  to  it  from  MYCIN.  Thus, 
EMYCIN  does  not  support  the  same  quasi -Engl i sh  rule  specification  nor  the 
checks  for  rule  consistency  that  were  features  of  TEIRESIAS' s  interaction  with 
the  user.  Instead,  EMYCIN  has  a  high-level  structure  editor  for  modifying  its 
knowledge  base.  Within  the  editor,  rule  specifications  are  entered  in  the 
:  ISP  programming  language.  The  editor  provides  some  syntactic  and  lexical 
error  checking  and  correction  as  well  as  automatic  bookkeeping  for  the 
knowledge  base.  EMYCIN  thus  requires  greater  computer-related  skills  from  it, 
users  than  did  TEIRESIAS. 

The  facilities  carried  over  from  TEIRESIAS  include: 


a.  Interactive  tracing  and  debugging  of  the  reasoning  leading  to 
concl usi ons . 

b.  A"  interface  to  the  performance  system's  explanation  facility  to 
obtain  summary  explanations  of  reasoning. 

c.  Automatic  testing  and  comparison  of  stored  case  data  and  results 
against  those  produced  by  a  modified  knowledge  base. 

In  general,  I TE  support  in  EMYClN  is  more  passive  than  that  in  TEIUIS1AS. 
placing  most  of  the  responsibility  for  initiative  in  the  user’s  hands. 

ROGET.  ROGET  (Bennett,  1983)  is  an  ongoing,  experimental  effort  to  provide 
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extensive  support  for  ITE  in  the  EMYCIN  system.  ROGET  is  unusual  in  that  it 
is  intended  to  consult  with  the  user  about  how  to  design  an  expert  system  for 
his  domain.  ROGET  applies  meta-knowledge  about  how  automated  consultants  for 
other  problem  domains  have  been  implemented  within  the  EMYCIN  architecture. 
This  experience  is  encoded  both  as  specific  facts  about  the  other 
implementations  and  as  generalized  concepts  and  rules  inferred  from  the 
specific  facts.  ROGET  thus  embodies  a  characterization  of  the  class  of 
problems  to  which  EMYCIN  is  applicable. 

ROGET  aoplies  a  set  of  dialogue  management  rules  to  control  a  question 
and  answer  dialogue  with  its  user.  The  dialogue  guides  the  user  through 
several  phases  of  description  of  the  desired  system.  The  initial  questions 
ask  about  the  purpose  of  the  system,  the  general  features  of  the  knowledge  it 
must  embody,  and  the  data  It  will  operate  on.  Examples  from  EMYCIN  and  other 
expert  systems  are  used  to  illustrate  the  classifications  the  user  can 
specify.  Using  these  data,  ROGET  generates  a  probabilistic  conclusion  about 
whether  EMYCIN  is  suited  to  the  task  and.  If  so,  what  time  and  resource 
expenditures  the  user  should  expect.  It  is  able  to  use  a  trace  of  its  own 
rules  to  explain  its  conclusions  to  the  user. 

In  the  next  phase  of  the  dialogue,  ROGET  guides  the  user  in  formulating 
the  abstract  conceptual  structure  of  the  task.  Again,  it  uses  its  knowledge 
of  other  systems  to  iTfustrate  possible  answers.  A  conceptual  structure, 
analogous  to  the  inference  models  of  PROSPECTOR/KAS,  describes  a  generic  model 
of  problem  solving  for  a  class  of  tasks  (e.g.,  clinical  classification). 

ROGET  uses  the  abstract  conceptual  structure  it  elicits  to  guide  a  dialogue 
elaborating  specific  models  for  the  user's  domain.  This  corresponds  to  the 
elicitation  of  taxonomic  and  inference  networks  in  PROSPECTOR/KAS. 

The  features  of  ROGET  are  only  partially  implemented  and  there  are  no 
real  data  on  the  effectiveness  of  the  system.  Even  so,  ROGET  is  a  significant 
new  step  toward  support  of  ITE.  It  is  the  first  system  to  provide  assistance 
in  initial  problem  formulation  (part  of  the  Definition  objective)  that  is 
fully  integrated  with  elaboration  and  refinement  of  the  formulation.  It  does 
this  using  its  knowledge  about  the  host  system  and  about  the  characteristics 
of  specific  types  of  problem  domains  implemented  in  the  host  system.  ROGET  is 
oriented  toward  use  by  a  KE;  however,  once  the  initial  characterization  and 
conceptual  structure  have  been  entered  by  the  KE  (based  on  interactions  with  a 
DE ) ,  it  should  be  possible  for  a  trained  DE  to  use  the  system  to  further 
elaborate  the  structure  with  only  intermittent  assistance  from  the  KE. 

The  approach  taken  in  ROGET  suggests  that  a  system  providing  assistance 
in  early  knowledge  acquisition  requires  historical  and  pragmatic  knowledge 
about  its  own  structure  and  about  the  class  of  problems  that  it  can  address. 

F XPERT .  r XPERT  (Weiss  and  Kulikowskl,  1979),  like  EMYCIN  and  PROSPECTOR/KAS, 
is  a  skeletal  system  for  building  consultation  systems  that  help  solve 
classification  problems. 

EXPERT’S  support  for  ITE  is  limited  to  testing  and  debugging  new 
knowledge.  Adding  new  knowledge  is  a  prograirmlng-like  activity  that  occurs 
with  no  direct  interface  to  the  performance  system.  Instead  the  user  mus»- 
employ  a  standard  text  editor  to  describe  the  knowledge  in  a  rigid,  special 
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purpose  language.  The  text  file  is  then  submitted  to  a  compiler  which  detects 
syntax  errors  and  generates  an  executable  internal  representation  of  the 
knowledge  base.  This  base  then  can  be  utilized  by  the  EXPERT  performance 
system.  Knowledge  acquisition  therefore  requires  users  trained  both  on  the 
system's  problem-solving  abstractions  and  use  of  general-purpose  computer 
systems. 

Testing  and  debugging  facilities  permit  tracing  the  operation  of  rules  on 
a  case  and  comparing  the  results  produced  by  different  versions  of  a  knowledge 
base  over  a  set  of  cases.  The  emphasis  is  on  design  of  inference  models  by 
debugging  and  refinement,  with  Initiative  strongly  vested  with  the  user. 
However,  the  need  to  use  an  external  editor  and  operating  system  facilities  to 
manage  knowledge  base  maintenance  assigns  greater  incidental  bookkeeping 
responsibilities  to  the  user  than  is  coronon  in  similar  skeletal  systems. 

AGE.  AGE  (“Attempt  to  GEneralize")  (Nii  and  Aiello,  1979)  is  a  system  for 
designing  and  building  expert  systems.  It  bridges  the  gap  between  skeletal 
systems  like  EMYCIN  and  specialized  AI  programing  languages.  AGE  aids  its 
user  in  selecting  and  configuring  control  formalisms  for  the  expert  system  the 
user  intends  to  build.  Ultima*ely,  the  user  must  encode  the  domain  knowledge 
into  the  selected  formalisms  using  the  LISP  programming  language.  However, 

AGE  provides  a  software  library  comprising  a  rule  interpreter,  rule  tracer, 
and  explanation  modules  as  well  as  the  representation  formalisms.  The  user 
has  access  to  these  through  an  intelligent  front-end  that  performs  automatic 
bookkeeping  and  provides  active  guidance  during  the  design  and  implementation 
of  the  user's  system. 

AGE  conducts  a  dialogue  generated  from  templates  to  elicit  a  design  from 
the  user.  The  user  predominantly  comnunicates  through  menu  choices,  with  text 
input  limited  to  labels  the  user  provides  for  components  of  the  design. 
Dialogue  management  is  based  on  prestored  control  knowledge  about  sequences  of 
archi tectural  decisions  and  implementation  activities  that  realize  the 
alternative  architectures.  At  decision  points,  AGE  offers  the  user  choices 
and  can  provide  advice  based  on  its  control  knowledge.  Using  a  command 
language,  the  user  can  escape  from  guided  design  elicitation  to  work  on  the 
design  and  implementation  in  any  order  he  wishes.  Subsequently,  he  can  return 
to  guided  design,  since  AGE  maintains  a  record  of  progress  within  both  the 
guided  and  unguided  modes. 

Despite  the  user-friendly  features  of  its  front-end,  AGE  is  strictly  a 
tool  for  Li!!1  knowledgeable  KE.  In  addition  to  LISP  programming  ability,  the 
user  must  learn  to  understand  and  use  the  technical  terminology  AGE  uses  to 
reference  the  software  models  it  includes.  ITE  support  is  limited  to  design 
and  implementation,  with  no  facilities  for  assisting  the  KE  in  interacting 
with  the  DE  to  acquire  domain  knowledge. 

GODDESS.  The  GODDESS  system  (Pearl,  Leal,  and  Saleh,  1982)  is  an  integrated 
knowledge  acquisition  and  decision  aiding  system.  The  performance  component 
of  GODDESS  is  not  an  expert  system,  but  we  include  the  system  in  our 
discussion  because  it  draws  on  AI  research  and  emphasizes  support  for  ITE. 

GODDESS  guides  a  naive  computer  user  in  the  elaboration  of  considerations 
underlying  an  impending  decision.  The  approach  depends  completely  on 
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syntactic  analysis  of  a  formalized  representation  of  the  user's  input 
accompanied  by  quantitative  evaluations  of  preference  and  confidence  judgments 
associated  with  elements  In  the  representation.  GODDESS  can  reconmend  a 
decision  based  on  the  user's  input,  but  more  importantly  provides  an 
interactive  capability  for  the  user  to  perform  a  sensitivity  analysis— to 
determine  the  criticality  of  particular  factors  and  values  in  determining  the 
recommended  decision. 

GODDESS  uses  a  simple  generic  representation  of  causal  models  for  the 
user's  input.  This  representation  is  an  AND/OR  graph  of  goals  and  actions 
derived  from  that  used  in  the  STRIPS  (Fikes  and  Nilsson,  1971)  system. 

Dialogue  management  is  based  on  the  structure  of  this  representation.  It 
starts  with  the  user's  ultimate  goal  and  proceeds  backward  iteratively  to 
elicit  subgoals,  alternative  actions  to  achieve  those  subgoals,  and 
preeondi tions  for  performing  those  actions.  As  each  action  is  elaborated  the 
user  estimates  its  cost  and  its  likelihood  for  causing  its  immediate  goal  to 
be  achieved.  The  order  in  which  subgoals  (preconditions)  are  pursued  in  the 
dialogue  is  determined  by  GODDESS  using  user-supplied  judgments  of 
cri tical i ty . 

The  user  interface  in  GODDESS  is  rigid  and  permits  no  user  initiative. 
Since  the  system  has  no  knowledge  of  any  task  domain  and  does  not  develop  a 
hierarchy  of  primitive  concepts,  it  must  treat  all  user  descriptions  of  goals 
and  actions  as  unparsed  character  strings.  This  limits  its  ability  to  check 
the  consistency  of  that  information.  The  developers  consciously  sacrificed 
this  capability  for  generality  of  applicability  and  availability  to  untrained 
users  with  a  wide  range  of  decision  problem  types. 

ROSIE.  ROSIE  (Rule-Oriented  System  for  Implementing  Expertise)  (Fain,  et  al . , 
1981)  is  a  high-level  programming  language  specially  designed  to  support  tKe 
implementation  of  expert  systems.  It  has  a  relatively  flexible  English-like 
syntax  and  a  variety  of  language  constructs  that  support  rale-based 
programning.  However,  it  does  not  include  an  integrated  design  for  knowledge 
representation  and  problem-solving  control  structures,  such  as  the  fixed 
design  of  EMYCIN  or  the  alternative  designs  of  AGE.  Instead,  the  ROSIE  user 
must  construct  these  manually  at  his  own  initiative  using  the  language’s 
constructs.  These  constructs  are  intended  to  support  transparent 
implementation  of  a  variety  of  problem-solving  archi tectures. 

The  ROS  IF.  user  interface  includes  the  features  Al  languages  such  as  LISP 
typically  offer:  interactive  editing.  Interactive  debugging  using  traces  and 
break  points,  and  automatic  file  management.  However,  these  are  oriented 
toward  the  level  of  abstraction  of  language  constructs,  not  that  of  the  user's 
design.  Thus,  ROSIE  provides  little  direct  support  for  ITE  except  insofar  as 
it  allows  the  user  to  program  using  the  same  concepts  and  English-like 
expression  of  rules  as  he  uses  to  describe  task  expertise.  ROSIE' s  appeal  to 
the  user  is  thus  not  its  support  for  ITE,  but  in  the  flexibility  it  affords 
for  implementing  system  capabilities  not  available  in  existing  skeletal 
systems  and  in  the  ease  it  affords  for  expressing  knowledge  and  rules. 
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CONCLUSIONS 

The  majority  of  efforts  that  have  implemented  knowledge-based  systems  for 
specific  problem  applications  have  included  some  attention  to  the  support  of 
ITE.  In  most  cases,  the  support  has  evolved  from  that  typically  available  in 
AI  programming  systems  for  interactive  testing  and  debugging.  The 
specializations  of  these  facilities  for  knowledge-based  systems  include 
manipulation  of  more  abstract  representations  of  knowledge  and  control  arid  of 
configurations  of  test  data  that  are  meaningful  for  the  application.  Support 
for  the  acquisition  of  new  knowledge  has  been  added  to  assist  incremental 
extension  and  refinement  of  expert  knowledge  bases.  TEIRESIAS  and  KAS  have 
explicitly  addressed  knowledge  acquisition  support  by  taking  an  active  role  in 
the  knowledge  elicitation  process.  To  do  so,  they  use  knowledge  about  the 
architecture  of  their  host  system,  about  specific  features  of  the  domain 
knowledge,  and  about  characteri sties  of  prior  knowledge  acquisition 
interactions. 

Skeletal  systems  and  other  generic  tools  for  building  knowledge-based 
systems  have  included  genera1 i zati ons  of  the  I TF  support  found  in  the 
application-specific  systems.  The  ITT  support  in  generic  systems  applies 
across  most  of  the  knowledge  base  development  process.  However,  because  it  is 
based  on  the  syntactic  structure  of  skeletal  system's  representati on 
formalism,  active  elicitation  is  more  mechanical  and  less  responsive. 

To  the  present  time,  support  for  ITE  has  been  oriented  toward  KEs  and  to 
a  lesser  extent  OEs  knowledgeable  about  formalisms  of  the  host  system.  Thus, 
i)F  use  has  been  limited  to  systems  that  are  already  stable  and  so  are  unlikely 
■'  require  fundamental  changes  of  capabilities,  design,  and  implementation  to 
accommodate  extensions  of  their  knowledge  base.  GODDESS,  the  one  system 
described  here  that  is  oriented  toward  use  by  naive  DEs,  is  not  a  true 
knowledge-based  system  in  that  it  has  no  understanding  of  the  knowledge  the 
user  encodes  beyond  its  formal  relational  structure.  For  true  knowl edge-based 
systems,  direct  use  by  DEs  for  ITE  appears  possible  only  with  special 
training.  The  amount  of  training  required  seems  to  bo  directly  proport  oal 
to  the  complexity  and  flexibility  of  the  system's  representation  formalisms 
and  inversely  proportional  to  the  system's  ability  to  apply  domain-dependent 
knowledge  in  managing  ITE  interactions. 
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SECTION  IV 

INSTRUCTIONAL  KNOWLEDGE  ACQUISITION  SYSTEM  CONCEPT 


In  this  section  we  present  our  concept  of  an  automated  knowledge 
acquisition  system  for  elicitation  of  information  from  a  DE  for  use  in  an 
instructional  system.  In  deriving  a  concept  for  automating  knowledge 
acquisition  and  reducing  dependence  on  KEs  for  training  system  development,  we 
considered  the  set  of  functions  performed  by  the  KE  (as  described  in  Section 
II).  Our  review  of  prior  work  on  aids  for  ITE  (described  in  Section  III) 
provided  data  on  the  current  state  of  the  art  in  systems  for  ITE,  the 
resources  required  to  achieve  various  capabilities,  and  the  feasibility  of 
automating  or  assisting  different  aspects  of  the  knowledge  engineering 
process.  Based  on  this  analysis,  we  considered  a  variety  of  possible  concepts 
for  tools  to  assist  the  DE  in  pre-fi 1 terlng  ideas  for  automated  training 
system  applications,  tools  to  assist  the  knowledge  engineer  in  the  system 
development  effort,  and  tools  to  assist  the  elicitation  of  knowledge  from  the 
DE.  The  set  of  specific  concepts  we  considered  included: 


a.  An  advisor  to  help  the  DE  determine  if  a  knowledge-based 
instructional  system  is  feasible  for  his  application 

b.  A  tool  recommender  to  advise  the  DE  what  training  approach  is 
tractable  and  what  instructional  technologies  can  be  brought  to  bear  on 
his  application. 

c.  A  knowledge  engineering  language  tailored  for  the  construction 
of  surrogate  instructor  training  systems. 

d.  A  data  base  completeness  checker  to  find  redundancies, 
inconsistencies,  and  knowledge  gaps  in  a  wide  range  of  application  areas. 

e.  A  test  case  generator  for  producing  and  managing  tests  of 
developed  systems. 

f.  A  system  to  elicit  from  the  DE  the  basic  declarative  concepts  in 
the  domain  and  the  relationships  among  them. 

g.  A  system  to  elicit  from  the  DE  an  expert  task  performance  model 
for  any  domain  ( 1 . e . ,  the  goals  and  procedures  used  to  perform  the  task 
correctly) . 

h.  A  system  to  elicit  an  expert  task  performance  model  for  tasks  in 
a  particular  domain  class  (e.g.,  tactics,  radar  operations)  that  would 
use  class-specific  knowledge  to  Intelligently  structure  interactions  with 
the  user. 

i.  A  system  to  elicit  the  deviations  in  an  expert  task  performance 
model  that  would  define  the  performance  model  (i.e.,  the  model  of  the 
types  of  incorrect  behaviors  the  trainee  might  choose  in  particular 
situations)  used  for  skill  diagnosis  during  training. 
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j.  A  system  to  elicit  the  rules  used  to  diagnose  trainee  behavior 
during  instruction  (e.g.,  IF  situation  AND  behavior  THEN  infer 
diagnosi s ) . 

k.  A  system  to  elicit  training  problems  or  situation  descriptions 
that  can  be  used  by  a  generator  to  produce  training  problems. 

To  evaluate  the  promise  of  these  concepts  in  meeting  the  objectives 
stated  by  NAV TRAFQUIPCEN ,  we  developed  a  set  of  criteria  for  system  utility 
against  which  these  concepts  could  be  weighed.  The  following  paragraphs 
describe  these  criteria,  and  the  subsequent  discussion  presents  our  concept  of 
an  automated  instructional  knowledge  acquisition  system  (IKAS)  that  emerged 
from  these  considerations. 

CRITERIA  FOR  EVALUATING  IKAS  CONCEPTS 

EVIDENCE  SUPPORTING  FEASIBILITY.  Perhaps  the  most  fundamental  criterion  for 
evaluating  the  suitability  of  a  concept  is  whether  or  not  its  implementation 
appears  to  be  feasible,  given  current  technology.  Judgments  of  the 
feasibility  of  <1  concept  depend  on  several  types  of  evidence.  Precedents  from 
prior  research  indicate  whether  the  knowledge  and  methods  to  perform  the 
function  under  consideration  ran  be  sufficiently  well  described  to  support  an 
implement  tirn.  Discussions  with  domain  experts  can  indicate  the  extent  of 
their  knowleoge,  their  ability  to  articulate  it,  arid  their  ability  to  engage 
in  particular  types  of  human-machine  dialogues  about  that  knowledge.  In 
general ,  the  feasibilit'-  of  realizing  a  particular  IKAS  architecture  varies 
directly  as  a  function  of  the  scope  and  complexity  of  the  concept. 

EASE  Of  IMPLEMENTATION.  A  set  of  IKAS  concepts  judged  feasible  may  vary  on 
the  anticipated  time  and  effort  required  to  implement  them.  Implementation 
cime  affects  the  cost  of  developing  a  system  and  therefore  influences  the 
evaluation  of  cost  against  expected  benefit.  A  significant  portion  of 
development  time  for  a  specific  IKAS  depends  on  che  time  required  to  build  .md 
stabilize  the  host  instructional  system  that  the  IKAS  will  serve.  Another 
determinant  is  the  scope  and  novelty  of  the  targeted  IKAS  capabilities:  the 
greater  the  anticipated  difficulty  in  implementing  a  capability,  the  greater 
the  expected  implementation  time.  Novelty  of  a  concept  can  be  assessed  by 
contras*  lo  prior  efforts  it  developing  knowledge  engineering  aids.  These 
prior  efforts  also  provide  a  source  of  data  for  anticipating  expected 
development  times  and  resource  requirements. 

REDUCTION  OF  Dr.  TIME-ON-TASK .  One  measure  of  the  value  of  an  IKAS  is  the 
extent,  to  which  it  reduces  the  time  OEs  must  invest  to  assist  in  the 
development  of  an  instructional  system.  If  an  estimate  can  be  made  of  the 
increased  efficiency  of  the  IKAS  over  manual  knowledge  acquisition,  savings  of 
BE  time  can  be  estimated  by  considering  the  relative  proportion  of  overall 
system  development  activities  involving  the  DE  for  which  the  IKAS  will  be 
used. 

REDUCTION  OF  KF  TIME -ON -TASK.  The  NAVTRAEQUIPCEN  statement  of  work  emphasized 
the  Inherent  value  of  reduction  of  the  KE ’ s  time  and  involvement  in  the  system 
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development  process.  The  cost  and  scarcity  of  skilled  expert  systems  builders 
perhaps  makes  this  criterion  an  important  consideration  in  attempting  to 
reduce  costs.  While  the  development  of  automated  capabilities  for  acquiring 
instructional  knowledge  should  reduce  the  requisite  involvement  of  the  KE  on 
that  portion  of  the  system  development  tasks,  the  presence  of  the  system  will 
most  likely  entail  other,  new  activities  for  the  KE.  For  example,  the  KE 
might  need  to  use  the  IKAS  himself  to  review  the  results  of  DE-machine 
interactions  and  obtain  knowledge  he  needs  to  perform  other  tasks.  An 
understanding  of  the  entire  system-building  process  (see  Section  II)  is 
critical  in  applying  this  criterion. 

EVIDENCE  FOR  USER  REQUIREMENT.  This  criterion  reflects  the  extent  to  which  an 
IKAS  is  perceived  by  potential  users  (both  DEs  and  KFs)  as  necessary  to 
supplement  or  replace  manual  methods.  Even  if  the  IKAS  does  not  reduce  time- 
on-task  for  either  DE  or  KE,  it  may  be  required  to  alleviate  the  difficulties 
of  scheduling  face-to-face  interactions  between  DE  and  KE.  DE  participation 
may  not  be  possible  unless  it  can  be  done  on  a  spontaneous,  opportunistic 
basis  proximate  to  the  DE  work  place.  Inputs  from  DEs  and  system  builders  are 
instrumental  for  determining  whether  there  are  user  requirements  for  an  IKAS 
in  order  to  build  a  knowledge-based  instructional  system  for  some  domain 
class.  Thus,  when  scheduling  and  geographic  constraints  restrict  the 
interactions  between  KE  and  DE,  the  IKAS  could  serve  as  an  institutional 
memory  to  record  for  subsequent  review  the  interactions  between  the  system  and 
the  DE  or  KE. 

INCREASED  FUNCTIONALITY.  Aside  from  possible  time  savings  and  increased 
convenience,  an  IKAS  may  improve  the  quality  of  the  knowledge  obtained  for  the 
instructional  system.  It  may  accomplish  this  through  the  use  of  more 
systematic  and  thorough  interviews  than  would  be  conducted  manually.  Relative 
functionality  can  be  estimated  by  comparing  the  capabilities  of  the  IKAS  to 
those  achieved  manually,  as  evidenced  by  prior  efforts.  There  is  an  obvious 
tradeoff  with  concept  feasibility  and  time-for-implementation  for  approaches 
that  attempt  to  signi f icantly  increase  functionality. 

GENERALITY  AND  SCOPE  OF  APPLICATION.  Generality  and  scope  refer  to  the 
breadth  of  applicability  of  the  IKAS  and  to  how  much  of  the  instruction 
typical  of  the  domain  can  be  accomplished  via  the  instructional  system.  They 
ilso  refer  to  how  much  of  the  knowledge  used  in  the  instructional  system  is 
obtained  by  the  IKAS  and  to  the  extent  of  IKAS's  role  in  the  life-cycle  of  the 
instructional  system.  Ideally,  when  the  domain  knowledge  is  constantly 
changing,  the  IKAS  should  play  a  role  in  eliciting  new  knowledge  from  the  DE 
to  update  to  an  existing  system. 

DE  BACKGROUND  AND  SKIIL  REQUIREMENTS.  A  significant  determinant  in  the 
success  and  utility  of  the  IKAS  will  be  the  extent  and  type  of  knowledge 
(apart  from  domain-specific  knowledge)  the  DE  needs  in  order  to  use  the  IKAS. 
ror  example,  it  seems  likely  that  an  IKAS  design  will  need  to  assume  its  user 
has  some  computer  background  or  skills.  Alternative  concepts  may  vary  in  the 
degree  to  which  other  special  knowledge--about.  learning  and  instruction  and 
about  knowledge-based  systems  techno! ogy-- 1 s  needed  by  their  users.  The 
number  of  DEs  for  any  problem  domain  who  are  potential  users  of  the  IKAS 
decreases  as  requirements  for  special  knowledge  increase.  This  factor  can 
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characteristics  of  and  procedures  required  by  different  specific  surface-  or 
air-search  radars. 

Restriction  of  the  scope  of  the  I K AS  in  this  fashion  will  permit  the  use 
of  semantic  and  pragmatic  knowledge  of  the  problem  class  for  effective 
management  of  knowledge  acquisition  interactions .  Our  concept  therefore 
includes  the  use  of  cl ass-generi c  knowledge  by  the  IK AS  to  guide  interactions 
with  the  user.  The  alaiTTty  to  use  knowledge  about  a  class  of  tasks  to  guide 
the  acquisition  of  instructional  knowledge  about  specific  tasks  in  that  class 
imposes  two  important  requirements  on  the  nature  of  the  I K AS .  First,  the 
design  of  the  instructional  system  must  be  stable,  so  the  knowledge  of 
structure  and  capabilities  used  by  IKAS  does  not  become  obsolete  and 
inappropriate.  Second,  the  development  of  the  IKAS  for  a  class  of  tasks 
depends  on  at  least  a  partial  implementation  of  one  application  within  that 
class.  This  implementation  would  assist  the  process  of  identi fication  of 
class-generic  knowledge  and  structure  using  general  concepts  induced  from  the 
specific  task  under  study.  The  resulting  IKAS  would  be  capable  of  supporting 
the  development  of  the  identical  i nstruc ti onal  system  for  other  members  of  the 
domain  class. 

The  following  section  discusses  three  promising  alternative  realizations 
of  the  IKAS  concept.  The  alternatives  were  selected  by  considering  the 
criteria  described  above  and  assessing  the  cost  and  expected  benefits  of 
development  of  the  concepts.  We  perceive  tradeoffs  among  the  criteria  as 
applied  to  the  alternatives  that  prevent  us  from  identifying  any  one  of  them 
as  best  from  a  technical  perspective.  However,  each  of  them  appears  to 
satisfy  NAVTRAEQUIPCEN 1  s  stated  objectives  and  entail  a  feasible  set  of  system- 
capabilities  that  extend  the  technology  currently  available  to  the  developers 
of  computer-based  training  systems. 

Al.  TERN  AT  I VF  1:  AM  Aid  FOR  THF  SPECIFICATION  OF  PERFORMANCE  MODELS.  Expert 
systems'-knowledge-based  systems  designed  to  act  as  consultants  for  domain 
spec  i  a  I  i  sts--embody  a  competence  model  ,  the  knowledge  that  oujjhjt  to  be  used  in 
jirob'  >m-sol ving  situations.  XnowTedqe-based  instruct i onal  systems  (KGIS),  if 
they  am  to  implement  performance  diagnosis  for  trainee  evaluation  or 
selection  of  i nstructional  TrTte'rventTons  ,~~al  so  need  performance  models.  These 
models  contain  representations  of  the  knowledge  learners  may  actually  apply  in 
situations.  Performance  models  describe  errorful  or  suboptimal  behavior 
relative  to  the  competence  model.  While  expert  systems  need  to  generate 
actual  competent  problem-solving  behavior,  some  KBISs  have  been  able  to 
perform  performance  diagnosis  with  descriptive,  less  complete  competence  and 
performance  models,  different  from  those  employed  in  expert  systems.  As  a 
consequence,  constructing  a  KB  I S  can  have  substantially  different  requirements 
for  knowledge  acquisition  than  an  expert  consultation  system  (Brown,  1977 ; 

(11  ancey ,  1981 ) . 

One  alternative  IKAS  concept  focuses  on  the  acquisition  of  the  knowledge 
necessary  to  construct  performance  models  in  a  diagnosis  module  of  a  KBIS. 

This  knowledge  includes: 


a.  Descriptions  of  non-expert  behavior  (errors,  parametric 
variations)  in  the  domain 
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b.  Characterizations  of  situations  that  <yv  '•-•  non-expert  behavior 
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defense  tactics'),  it  is  not  clear  at  what  level  of  abstraction  (ships 
within  classes,  classes  within  platform  types,  types  within  mission 
areas)  generality  can  be  achieved  for  this  IKAS  alternative. 

c.  It  is  not  evident  whether  this  system  would  reduce  the  DE '  s 
time-on-task  during  the  development  of  a  kBIS.  The  DF.  would  be  involved 
in  both  the  manually-based  development  of  the  competence  model  and  the 
IKAS-based  development  of  t.ie  performance  model.  It  is  possible  that  the 
duration  of  both  could  be  reduced  over  a  fully  manual  approach  because 
the  IKAS  wi r  ,,e  able  to  use  existing  class-generic  krowledge  in  both 
types  of  models.  The  KE  1  s  time-on-task  would  be  reduced  to  the  degree 
the  IKAS  replaces  him  in  eliciting  performance  models  from  the  DE.  It  is 
likely  that  these  savings  would  be  substantial  because  the  knowledge 
embodied  by  performance  models  is  typically  many  times  greater  than  that 
of  the  correspond! ng  competence  model.  However,  total  time  savings  would 
undoubtedly  be  offset  to  a  degree  by  the  KE 1 s  need  to  familiarize  himself 
with  the  performance  models.  Although  a  key  assumption  of  the  general 
IKAS  concept  is  that  the  KE  will  not  undertake  modifications  of  the  KBIS 
archi tec  tore ,  there  will  certainly  be  some  need  for  the  KE  and  DE  to 
refine  and  revise  the  knowledge  acquired  using  the  IKAS.  To  do  so,  the 
KE  will  need  to  spend  time  acquainting  himself  with  that  knowledge. 

J.  This  IKAS  concept  will  require  some  training  for  the  DF  so  that 
lie  can  use  the  system  interface  and  understand  the  knowledge 
representation  used  in  the  KBIS.  This  background  will  be  necessary  to 
allow  the  DE  to  engage  in  meaningful  dialogues  and  to  provide  useful 
information  about  potential  variations  from  competent  behavior  and 
situational  parameters  that  affect  performance.  An  important  advantage 
of  this  alternative  is  that  such  training  can  be  TncTdenta  ll  y  embedded  in 
tFe  interactions  "between  TTieTF^ancf  tiTTIurfh^  HeveTonme'nt  of  the” 
competence  model .  Those  interactions  wiTFThtrd'duce’  the"  representation 
din  ceptTTo  "the  OE.  curther,  if  the  KE  works  with  the  DE  while  using  the 
IKAS  interface  to  develop  and  test  the  competence  model,  then  the  DF 
should  become  sufficiently  familiar  with  the  IKAS  that  his  later  use  of 
the  system  can  be  supported  largely  by  reference  manuals,  with  little  or 
no  need  for  formal  tutorials.  Thus,  while  this  IKAS  alternative  requires 
background  arid  skills  for  its  use,  we  anticipate  very  little  (indicated 
training  will  be  needed  for  users  to  acquire  them. 

ALTERNATIVE  ?:  MODIFICATION  OF  AN  OPPONENT  SIMULATION  FOR  TACTICAL 
TRAINING.  The  second  alternative  for  IKAS  concept  development  addresses  a 
training  capability  that  does  not  depend  directly  on  performance  diagnosis  and 
the  performance  models  it  requires.  The  capability  involves  providing  the 
capability  for  the  DF  or  instructor  to  define  and  sequence  the  set  of  problem 
situations  to  be  presented  to  the  learner.  Providing  this  capability  to  alter 
the  training  system  would  allow  the  instructor  to  develop  curricula,  monitor 
and  participate  in  training  exercises,  and  control  the  course  of  instruction. 
The  specific  example  of  this  capability  we  will  consider  here  entails  the 
modification  of  the  behavior  of  an  automated  opponent  in  a  tactical  training 
simulation.  (However,  tho^p  are  analogous  capabilities  in  other  domai ns- - for 
'-•v.riple,  the  manner  in  which  a  piece  of  equipment  will  behave  in  a  maintenance 
training  simulator.) 
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The  system  concept  assumes  that  the  opponent  simulation  is  based  on  a 
rule-based  representation  of  the  actions  and  procedures  the  opponent  should 
execute  in  tactical  situations.  Further,  it  assumes  that  prior  to  use  of  the 
I K AS ,  the  definitions  of  rules,  high-level  procedures,  and  primitive  actions 
are  sufficiently  complete  that  the  opponent  simulation  is  operational.  At 
this  point  during  system  development,  its  behavior  will  probably  represent  an 
initial  formulation  of  a  skilled  opponent.  Use  of  the  IKAS  would  allow  the  DE 
to  specify  variations  of  rules  and  procedures  governing  the  opponent's 
behavior.  Such  modifications  would  produce  alternative,  perhaps  suboptimal, 
opponent  behiviors--performance  models  for  the  opponent,  as  opposed  to 
performance  models  of  the  learner  as  in  Alternative  1.  The  variations  might 
include  both  structural  and  parametric  changes.  Such  modifications  might  be 
intended  to  achieve  a  number  of  goals — variation  in  skill  levels,  use  or 
avoidance  of  particular  tactics  or  systems,  or  the  creation  of  particular 
tactical  situations  for  the  trainer  to  respond  to.  Such  variants  could  be 
used  for  a  number  of  pedagogical  purposes,  depending  on  the  training  goals  of 
the  instructor. 

The  domain  of  this  system  concept  is  similar  to  that  pursued  in  previous 
research  sponsored  by  NAVTRAEQUIPCEN  ( AT perovi tch ,  1982).  That  work,  however, 
focused  on  the  acquisition  of  an  initial  set  of  procedures  and  presented  the 
user  with  a  prograrrini ng-1  ike  task  requiring  significant  knowledge  and  skill. 
The  approach  advocated  here  uses  the  IKAS  to  aid  the  RE  in  specifying 
modifications  of  knowledge  the  system  already  contains.  That  knowledge, 
including  class-generic  knowledge  about  the  characteristics  of  variations 
obtained  in  the  manual  development  of  a  first  system,  can  be  used  by  the  IKAS 
to  provide  greater  system  initiative  in  suggesting  what  variations  might  be 
generated  and  what  sort  of  modifications  might  achieve  them.  In  addition, 
because  modifications  are  always  made  to  an  operational  set  of  rules  and 
procedures,  the  DE  can  develop  them  incrementally  and  exercise  them  in  the 
KB  I S  through  an  interface  to  the  IKAS.  The  DE  can  thus  approach  the  task 
empirically  without  a  fully  developed  abstract  understanding  of  the  opponent 
simul ation . 

This  alternative  has  the  following  character! sti cs : 


a.  The  relatively  limited  scope  of  the  knowledge  to  be  acquired  by 
the  IKAS  in  this  alternative  argues  for  its  feasibility.  The  system 
concept  reduces  the  complexity  of  the  task  of  acquiring  procedural 
knowledge  by  limiting  IKAS  activities  to  secondary  elaborations,  thereby 
avoiding  initial  formulation  by  open-ended  specification.  However,  we  do 
not  perceive  significant  differences  in  feasibility  and  risk  between  this 
concept  and  Alternative  1.  This  alternative  does  have  greater  potential 
for  increasing  functionality  of  knowledge  acquisition  beyond  that  of 
manual  methods.  The  ability  to  test  modifications  by  executing  the 
entire  K3IS  could  increase  the  DE’s  productivity  and  decrease  the  need 
for  KE  involvement  in  post  hoc  knowledge-refinement.  The  generality  of 
this  alternative  is  limited  to  a  class  of  tactical  problems  with  a  common 
semantics  for  situation  and  action  descriptions:  tactics  problems 
involving  physical  threats  and  the  movement  of  platforms. 

b.  This  alternative  may  have  a  more  rapid  implementation  time 
compared  to  Alternative  1  if  it  were  coordinated  with  recent  or  ongoing 
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efforts  to  develop  intelligent  opponents  for  simul ator-based  training.^ 

In  this  case,  at  least  some  of  the  effort  required  to  implement  a  first 
KBIS  in  the  domain  class  could  be  circumvented.  Coordination  would 
depend,  from  a  technical  standpoint,  on  the  ability  of  the 
representations  used  in  the  existing  efforts  to  support  incremental 
elaboration.  The  implementation  time  advantage  may  be  augmented  by  the 
relatively  smaller  scope  of  the  knowledge  to  be  acquired  by  the  IKAS.  We 
also  perceive  a  greater  requirement  for  completeness  in  specifying 
competence  models  (as  in  Alternative  1)  than  in  specifying  alternative 
models  of  opponent  behavior  vis  £  vis  the  instructional  capabilities  that 
might  use  those  models  within- a  RrTTT 

c.  The  implications  of  this  system  concept  for  Of  and  KE  time-on- 
task  are  similar  to  those  for  Alternative  1.  DE  time  should  be 
comparable  to  that  in  manual  knowledge  acquisition,  while  KE  time  should 
he  reduced  by  the  DE ' s  use  of  the  IKAS,  with  the  net  savings  affected  by 
the  extent  to  which  the  KE  must  engage  in  compensating  activities.  The 
relative  savings  depend  on  what  other  capabilities  are  included  in  the 
KBIS  (e.g.,  performance  diagnosis  and  instructional  interventions) . 

Since  this  alternative  for  IKAS  development  does  not  affect  acquisition 
of  knowledge  for  those  capabilities,  the  overall  role  of  the  KE  in 
knowledge  acquisition  could  remain  significant. 

d.  This  system  would  require  DEs  to  have  a  significant 
understanding  of  the  knowledge  representation  formalisms  used  in  the 
KBIS.  The  predominantly  procedural  nature  of  the  formalisms  for 
implementing  an  opponent  simulation  (as  opposed  to  descriptions  of  errors 
and  situations  in  Alternative  1),  would  most  likely  require  greater  skill 
and  knowledge  of  the  IKAS  users.  As  in  Alternative  1,  however,  the  prior 
person-to-person  interaction  between  the  KE  and  DE  to  develop  the  initial 
opponent  model  should  provide  the  DE  with  the  requisite  knowledge  for 
using  the  IKAS. 

AlTFRNATtVF  .5:  CONSTRUCTION  Of  K NOW  1  EDGE  BASES  FOR  INSTRUCTION  ON  DOMAIN 
FACTS.  The  third  alternative  we  describe  emerged  from  interviews  with  system 
builders  and  DFs  at  the-  Navy  Personnel  Research  H  Development  Center  (NPRPf) 
(see  Section  V).  In  discussing  our  concept  for  a  class-generic  IKAS  with  thorn 
we  learned  that  an  NPROC  project  had  developed  an  experimental  prototype  of  a 
generic  KBIS  (McCandless,  1981;  Crawford  and  Hollan,  1983)  and  that  attempts 
to  have  DEs  implement  knowledge  bases  for  new  problem  domains  had  encountered 
difficulties.  They  believed  that  an  IKAS  for  the  system  is  necessary  if  its 
application  to  other  domains  is  to  become  cost-effective. 

The  NPRDC  system  was  first  implemented  for  the  surface  warfare  domain. 


Efforts  we  have  identified  include  Contracts  N5133R-80-C-0079  (Naval 
training  equipment  Center)  and  N00014-82-C-0653  (Office  of  Naval  Research). 

I  he  Navy  Personnel  Research  &  Development  Center  has  also  proposed  research  on 
tactical  decision  making  training  that  might  encompass  the  use  of  opponent 
s irnui ati ons . 
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where  its  objective  w.:s  to  teach  the  concepts  ana  facts  Tactical  Action 
Officers  (TAO)  mast  memorize  in  order  to  perform  threat  identification, 
assessment,  and  counter-a's's  i  gnment.  We  will  therefore  refer  to  the  system  as 
the  Computer-based  Memorization  System  (CMS).  The  CMS  was  implemented  with  a 
highly  modular  arehi tecture.  A  semantic  network  is  used  to  represent  the 
domain  concepts  and  facts  in  a  relational  database.  The  database  describes 
taxonomies  of  objects  and  object  attributes  and  relates  particular  objects 
with  their  attributes.  Instruction  is  provided  by  a  set  of  seven  modular 
"games"  (e.g.,  "twenty  questions",  "concentration")  each  of  which 
independently  uses  the  semantic  network  to  generate  problems.  Generation 
depends  solely  on  the  network  structure,  not  the  domain  content.  Thus,  the 
CMS  architecture  is  actually  generic  to  any  problem  domu . n  for  which  concepts 
and  facts  can  be  descrioed  using  its  semantic  network  formalism  and  for  which 
memorization  of  facts  is  an  instructional  objective.  It  has  subsequently  been 
applied  at  MPRDC  to  knowledge  Oases  for  the  domains  of  EW,  ECCM,  and  sonar 
analysi s . 

The  current  procedures  for  creating  new  Knowledge  bases  for  the  CMS  are 
not  conducive  to  direct  use  by  DCs.  The  knowledge  base  is  created  using  an 
off-line  editor  and  "compiled"  for  use  in  the  CMS.  The  physical  Interface 
requires  thorough  now  1  <  dge  or  the  progranini  no  system  in  which  the  CMS  is 
implemented.  The  editor  is  separate  from  the  CMS  and  places  the  knowledge¬ 
base  development  .co  ■  ,s  solely  under  the  user's  initiative  with  no  conceptual 
support--for  example,  oy  using  existing  knowledge  bases  ana  data  about  their 
development.  All  suen  conceptual  support  must  bo  obtained  externally  by  the 
user.  The  lack  of  conceptual  support  has  manifested  itself  in  the  ability  of 
one  DE  to  master  the  physical  interface  and  create  a  syntactically  slid 
knowledge  ease,  which  nonetheless  provided  inadequate  behavior  in  the  CMS 
games  because  of  the  DE's  inability  to  anticipate  how  ♦'he  knowledge  in  the 
network  would  or  organized  and  used  to  generate  problems. 

Thus,  a  third  alternative  for  pursuin';  ]K AS  development  is  to  build  upon 
the  existing  CMS  work.  The  objectives  would  or  to  provide  conceptual  sup; ort 
and  a  more  ac. ess' bio  physical  interface  for  the  user.  rhis  alternative  has 
the  following  characteristics; 

a.  I'e  a.> ;  hi  1  i  tv  of  this  il  ♦■••miati  ve  is  higher  than  for  Alternatives 
1  and  ?.  'hr  •  >»  i  s  tern.;'  oi  a  generic  KB  IS  already  in  use  by  DCs  for 
knowledge  base  development  is  persuasive  evidence  for  the  tractability  of 
the  IK  An  cor.  sot. 

t).  The  generality  of  this  alternative  is  somewhat  different  from 
the  concept  or  generality  we  have  introduced  previously.  In  the 
preceding  discussion,  we  used  the  concept  of  a  class  of  domains,  to  which 
an  IK  AS  and  Kbit  would  apply.  Caen  member  of  a  class  would  have  a 
significant,  overlap  of  semantics  and  pragmatics  and  utilize  a  common  set 
of  instructional  methods.  The  fMS  approach  achieves  generality  by 
isolating  a  ; ype  uf  knowledge  that.  ir  a  part,  possibly  a  very  small  part, 


i 

Two  exceptions  are  games 


specific  to  the  TAD  domain  that  use  graphics. 
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of  the  knowledge  in  a  very  large  number  of  problem  domains  whose  other 
types  of  knowledge  may  be  very  different.  It  thus  achieves  generality 
across  domains  by  greatly  limiting  scope  within  each  domain.  That  is, 
the  system  would  address  only  factual,  conceptual  knowledge,  which  is 
only  a  small  part  of  what  constitutes  competence.  One  consequence  of 
this  approach  to  obtaining  generality  is  that  an  IKAS  for  the  CMS  will  be 
unable  to  use  class-generic  semantic  and  pragmatic  knowledge  for  dialogue 
management.  Instead,  dialogue  management  will  have  to  be  based  on  the 
syntax  of  the  network  representation  and  on  knowledge  of  how  game 
capabilities  access  the  knowledge  base.  Thus,  this  alternative  appears 
to  achieve  feasibility  and  generality  across  domains  at  the  expense  of 
limiting  the  power  of  IKAS  techniques  available  to  the  system  and  the 
completeness  of  the  knowledge  that  can  be  acquired. 

c.  The  implications  for  this  alternative  for  DE  time-on-task  are 
again  difficult  to  assess.  Within  the  scope  of  appl icabil i ty,  KE  time- 
on-task  would  be  greatly  reduced  since,  in  contrast  to  Alternatives  1  and 
2,  there  is  no  programmatic  requirement  for  KE  involvement  in  the 
knowledge  acquisition  process.  We  expect  however  that  some  ad^  hoc  KE 
involvement  for  consultation  and  tuning  would  be  required  in  most  cases. 

d.  As  an  exploration  of  IKAS  technology,  this  alternative  would 
involve  less  time  and  resources  than  Alternatives  1  and  2.  The  generic 
KB I S  exists  and  several  problem  domains  have  already  been  implemented 
using  it.  New  effort  could  be  focused  on  the  IKAS  architecture  and 
implementation,  unless  some  re-implementation  of  the  CMS  into  a  more 
tractable  software  environment  was  deemed  necessary  to  support  IKAS 
functionality.  That  the  scope  of  application  is  limited  to  acquiring 
only  declarative  knowledge  would  also  serve  to  control  costs  and 
implementation  time  for  this  concept. 

e.  Users  of  an  IKAS  for  the  CMS  would  need  to  have  an  understanding 
of  the  semantic  network  representation  and  its  relationship  to  the 
mechanisms  of  the  instructional  games.  This  understanding  would  have  to 
be  developed  by  either  explicit  training  or  by  training  embedded  in  the 
IKAS.  The  latter  training  might  utilize  examples  from  previously 
implemented  domains.  The  degree  of  initial  training  required  would  be 
reduced  if  the  interface  between  the  IKAS  and  CMS  facilitated  an 
empirical  approach  to  knowledge-base  development  (as  in  Alternative  2). 

In  such  an  approach,  QEs  could  readily  experiment  with  possible  knowledge 
formulations  using  the  CMS  games.  Because  the  knowledge  involved  is 
solely  declarative,  we  anticipate  that  the  physical  interface  to  the  IKAS 
would  require  computing  skills  similar  to  those  required  by  a  stand-alone 
word  processor,  in  contrast,  the  specification  of  procedural  knowledge, 
as  suggested  by  Alternative  2,  has  some  of  the  characteristics  of 
programing  and  would  potentially  require  more  sophisticated  computing 
skills. 
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SECTION  V 

I K AS  DESIGN  CONSTRAINTS 


In  this  section,  we  consider  further  constraints  on  the  design  of  an 
instructional  knowledge  acquisition  system  (TKAS).  In  the  previous  section, 
we  enumerated  several  <~riteria  for  evaluating  alternative  IKAS  concepts. 

These  criteria  reflected  high-level  technical  and  organizational  concerns. 

The  present  discussion  will  consider  issues  that  bear  upon  the  design  of  user 
interfaces  cor  IKAS  systems.  The  sets  of  issues  are  not  independent,  nor  is 
the  design  of  an  IKAS  user  interface  independent  of  the  purpose  and  features 
of  the  KBIS  it  serves.  Our  main  objective  for  this  section  is  to  identity  how 
rr  'ui rements  for  easy  usage  of  an  IKAS  influence  the  design  of  the  system  s 
user  interface. 

We  first  enumerate  imports  it  dimensions  of  design  variation  in  user- 
interfaces,  providing  for  each  some  background  discussion  of  the  current 
methods  and  techniques  associated  with  that  aspect  of  interface  design.  We 
then  discuss  the  design  requirements  in  the  contort  of  the  general  IKAS 
concept  and  its  three  alternative  realizations  presented  in  Section  IV.  i he 
evidence  supporting  our  conclusions  about  these  requirements  draws  upon  the 
literature  on  human  interface  design,  out  it  comes  primarily  from  statements 
elicited  from  a  sample  of  Navy  DEs,  builders  of  state-of-the-art  instructional 
technology,  and  developers  of  knowledge-based  systems.  We  then  present 
necomnendaticns,  which  are  based  on  assumptions  about  host  system  architecture 
and  capabilities,  for  formulating  a  user  interface  design  for  a  specific  IKAS. 

We  do  rot  present  here  a  general  review  of  the  growing  literature  on 
human  factors  in  the  design  of  interactive  computer  systems.  Such  a  review 
was  beyond  the  resources  and  scope  of  the  current  project.  In  any  case,  our 
view  of  that  literature  is  that  detailed,  substantive  conclusions  about 
effective  interface  design  are  specific  to  task  applications,  such  as  word 
processing  and  database  query.  The  interactive  computing  tasks  that  have  V •-"> 
studied  are  very  different  from  the  task  of  transferring  new  knowledge  to  a 
knowledge -based  system.  Hence,  specific  details  of  user  interface  design  used 
in  other  inter, active  computing  tasks  are  of  limited  use  in  designing  the 
interface  to  an  IKAS.  More  general  principles  r.f  design  of  the  type  we  will 
discuss  ire  the  i.oronon  lore  of  system  builders  and  are  difficult,  to  attribute 
to  any  one  effort.  They  arc  also  vague.  Tor  this  reason,  we  refer  to  them  as 
"issues"  of  design  instead  of  ar  principles  of  design. 

USER  INTERFACE  ISSUES  ANO  METHODS 

Our  user  interface  issues,  derived  from  discussions  with  Navy  DEs  and 
instructional  system  builders,  are  similar  to  others  identified  in  the 
’’iterature  on  interactive  computer  system  design  (see  recent  reports  by  Ramsey 
and  his  colleagues  [Ramsey,  Atwood,  and  Kirshbaum,  1978;  Ramsey  and  Atwood, 
1979]  for  a  bibliography  and  review  of  this  literature;  see  Martin  [1976]  and 
Simpson  [1982]  for  similar  classifications).  A  critical  feature  of  the  issues 
is  their  interdependence:  particular  interpretations  of  how  an  issue  affecfr 
design  constrain  interpretations  of  other  issues.  A  second  critical  feature 
is  that  the  methods  used  to  achieve  a  design  feature  associated  with  an  issue 
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are  generally  not  applicable  to  all  interface  designs.  Instead,  most  designs 
involve  hybrid  methods. 

MEDIA.  The  issues  surrounding  input/output  media  are  perhaps  the  most  obvious 
in  user  interface  design.  The  CRT  has  become  almost  a  universal  output 
medium,  but  issues  about  display  details — graphics,  screen  resolution,  color 
output,  multiple  displays — remain.  Other  output  media  currently  receiving 
attention  include  videodisc  and  computer-generated  voice  and  sound.  Input 
media  include  the  typewriter-like  keyboard  built  into  most  CRTs,  devices  for 
spatial  designation  and  manipulation— touch  panels,  light  pens,  bitpads, 
"mice,"  and  voice. 

A  major  rationale  for  designs  incorporating  multiple  media  has  been  a 
general  belief  that  distributing  interaction  functions  among  perceptually 
distinct  channels  is  beneficial.  There  are  more  specific  issues  as  well 
regarding  input  mode  and  the  organization  of  interaction  that  are  connected 
intimately  with  use  of  media.  Input  mode  may  scale  from  open  to  closed,  where 
open  denotes  responsibility  for  the  user  to  remember  and  structure  "Tegal " 
inputs  and  closed  denotes  the  system's  responsibility  to  continually  inform 
the  user  of  "legal"  inputs  which  he  may  select.  The  open  mode  of  input  is 
typically  accomplished  using  artificial  languages,  which  may  approach  natural 
language  in  their  flexibility  and  complexity.  Closed  mode  is  exemplified  by 
methods  such  as  menus  and  templates. 

Related  to  input  mode  is  the  organization  of  interaction.  At  one 
extreme,  interactions  can  be  organTzecTTi ngui stTcaTTy/  'They  occur  on  a 
dimension  of  time  and  use  linguistic  devices  for  reference  to  earlier  events. 
At  the  other  extreme,  interactions  can  be  organized  spatially.  They  reference 
objects  and  actions  by  direct  manipulation  of  the  structure  of  visual  models. 
The  tempiral  connection  between  action  and  effect  is  discarded  (at  least  in 
the  context  the  interface  provides  to  the  user). 

The  linguistic  organization  of  interaction  can  be  combined  with  both  open 
and  closed  input  mode.  For  example,  the  system  can  provide  its  output  as  text 
and  accept  inputs  as  command  language  inputs,  choices  from  a  set  of 
alternatives  via  a  keyboard,  or  choices  made  by  pointing  to  its  text  output. 
Many  systems  mix  both  input  modes.  Much  recent  interest  in  spatial 
organization  has  been  motivated  by  the  objective  of  making  the  closed  input 
mode  more  efficient.  However,  spatial  organization  is  also  compatible  with 
open  input  mode.  Image  editing  systems,  for  example,  accept  typed  inputs  or 
sequences  of  movements  and  button  signals  from  a  mouse. 

Hi  SPON'UV!  HI  ss.  The  physical  responsiveness  of  an  interactive  system  is  a 
common  issm*.  Poor  responsiveness  is  held  responsible  for  both  undermining 
motivation  and  directly  disrupting  the  user's  thought  processes;  either  case 
reduces  productivity.  On  the  other  hand,  it  is  possible  that  response  can  be 
too  fast  for  some  tasks  and  some  users  when  thoughtful  behavior  is  required. 
The  user  should  neither  feel  delayed  nor  pressured  by  the  system's 
responsiveness.  However,  in  practice  too  rapid  response  is  not  a  serious 
concern  of  system  designers. 

Poor  response  has  both  delay  and  variability  components.  It  is  commonly 
thought  that  extreme  variability  is  more  disrupting  than  mere  delay. 
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System  response  depends  on  the  capacity  and  loading  of  the  entire 
hardware-software  environment.  Ultimately  this  means  that  achieving  a  better 
responsi veness  may  entail  eliminating  possible  features  of  the  target  system. 
Unfortunately  it  is  usually  difficult  to  predict  system  responsiveness  prior 
to  implementation.  Thus,  one  method  for  addressing  system  responsiveness  is 
to  design  and  implement  features  in  independent  and  separable  modules  to 
permit  alternative  system  configurations. 

Anot''  -r  method  for  addressing  system  delay  is  to  signal  the  user  about 
expected  delays  of  response.  This  involves  endowing  the  system  with  knowledge 
about  itself  so  that  it  can  indicate  to  the  user  when  it  is  about  to  perform 
operations  that  involve  considerable  time. 

The  use  of  networks  to  gain  access  to  systems  on  which  the  user  software 
executes  complicates  the  responsiveness  issue.  Networks  introduce  delay  and 
variability  that  the  original  designers  and  implementors  can  only  roughly 
anticipate.  In  addition,  they  reduce  the  utility  of  signalling  methods,  since 
it  seems  likely  that  no  signalling  is  better  than  the  incorrect  and 
inconsistent  signalling  that  a  system  operating  across  a  network  can  provide. 

FLEXIBILITY.  Flexibility  involves  three  characteristics  of  an  interface: 


a.  Alternative  input  possibilities 
h.  Alternative  output  possibilities 
c.  Handling  of  errorful  inputs 

The  first  two  of  these  are  related  to  the  needs  and  preferences  of  different 
users  or  of  a  single  user  accessing  the  system  for  different  purposes.  One 
common  method  for  achieving  increased  flexibility  is  the  use  of  modifiable 
prof i les  that  allow  settings  of  switches.  These  switches  can  be  referenced  by 
the  user  interface  software  to  select "aTterna ti ve  behaviors.  Other  common 
method 5  depend  on  the  implemented  physical  media  and  input  mode.  One  extreme 
form  of  flexibility  allows  both  open  or  closed  input  specification  when 
possible.  Flexibility  in  open  input  is  increased  by  designing  a  language  with 
multiple  devices  for  expressing  the  same  message  to  the  system.  This  includes 
redundant  lexicon,  alternative  syntactic  constructs,  provision  of  defaults  for 
■  I i ded  input,  and  resolution  of  anaphoric  reference.  At  a  lower  level, 

? ■ecogni \ ion  of  partial  inputs  also  increases  flexibility.  For  closed  input, 
such  as  menus,  flexibility  is  enhanced  by  use  of  multiple  media-pointing 
devices,  keyboard  function  keys--  as  alternatives  for  selecting  among  the  same 
responses. 

A  system  in  which  errorful  inputs  cause  a  system  to  break-confronting 
the  user  with  the  interface  of  the  underlying  operating  system  and  requiring  a 
restart--is  called  brittle.  Systems  are  made  less  brittle  by  methods  such  as 
prevention  and  trappTng  of  events  that  cause  operating  system  interrupts  and 
by  other  "worst  case"  assumptions  in  input  handling  code.  More  advanced 
approaches  add  spelling  correction  (for  open  input  mode)  and  other  "do  what  I 
mean"  capabilities  for  interpreting  anomalous  inputs.  One  rationale  for  the 
use  of  closed  input  modes  is  that  they  are  a  relatively  simple  technique  for 
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making  a  system  more  robust. 

CONTEXT.  Preserving  and  providing  user  access  to  the  context  of  the  human- 
machine  interaction  is  important  for  providing  recovery  from  errors, 
especially  those  resulting  from  user  inputs  that  are  semantically  meaningful 
but  have  undesirable  consequences.  The  user  needs  to  be  able  to  recover  from 
such  errors  by  invoking  an  "undo"  command.  Preservation  of  context  is  also 
important  for  providing  capabilities  for  users  to  easily  interrupt  their  work 
session  and  resume  it  in  the  future.  Methods  for  achieving  this  capability 
are  typically  linked  to  conventions  for  process  and  file  manipulation  specific 
to  the  host  operating  system. 

User  access  to  context  is  important  in  conceptually  supporting  the  user 
in  complicated  activities  with  the  system.  The  simplest  type  of  context  is 
feedback  about  the  effects  of  user  inputs.  It  plays  a  role  both  in  learning 
to  use  a  system  and  in  preventing  the  propagation  of  semantic  errors  through  a 
sequence  of  events  that  makes  them  difficult  to  undo.  In  a  1 ingui stical ly- 
oriented  interface,  feedback  usually  comprises  descriptions  of  effects  or 
paraphrases  of  the  user's  input.  In  a  vi sual -spati al  interface,  feedback 
comprises  perceptible  changes  in  the  visual  model  displayed  to  the  user.  In 
the  latter  case,  the  association  between  previous  user  inputs  and  their 
results  is  more  difficult  to  represent,  since  the  model  typically  has  no 
natural  means  for  expressing  the  order  of  effects  on  it.  Thus,  visual -spatial 
organization  may  make  it  more  difficult  for  a  user  to  remember  or  reconstruct 
what  sequence  of  inputs  led  to  a  particular  situation.  On  the  other  hand, 
that  type  of  interface  does  make  the  net  of  all  effects  continually  available 
to  the  user,  whereas  in  a  linguistic  interface  the  capacity  of  the  display 
medium  limits  the  prior  history  that  is  immediately  available.  The 
traditional  form  of  interaction  history  for  linguistic  interfaces  is  a 
hardcopy  record.  Newer  methods  dependent  on  high-speed,  high-capacity 
displays  make  records  of  prior  interactions  available  via  interactive 
"browser"  facilities. 

Another  issue  involving  context  concerns  the  handling  of  multiple  and 
subsidiary  contexts.  Systems  for  complex  tasks  may  include  different  mode., 
for  different  activities.  The  scope  of  a  single  activity  may  be  too  great  to 
allow  all  relevant  information  to  be  displayed  simultaneously.  Under  such 
c  irc.umstanccs ,  the  user  can  lose  track  of  the  local  context  of  his 
infraction,  fine  approach  to  this  problem  is  to  always  provide  cues,  either 
linguistic  or  perceptual,  about  the  current  context  and  its  surrounding 
contexts.  An  additional  method  is  to  make  transitions  between  contexts  non¬ 
destructive  by  preserving  them  and  providing  mechanisms  for  returning  to  them. 
Such  facilities  can  include  user  accessible  information  about  suspended 
contexts  and  mechanisms  for  alerting  the  user  when  activities  in  other 
contexts  are  incomplete.  This  kind  of  active  monitoring  is  dependent  of  the 
interface's  use  of  knowledge  about  the  task  domain. 

USER  CONTROL.  User  control  refers  tc  the  system's  facilities  for  allowing  th*-> 
user  latitude  in  the  way  he  approaches  his  task.  It  overlaps  with  the  issue 
of  flexibility  regarding  control  over  how  to  enter  inputs.  More 
fundamental ly,  it  Includes  questions  about  the  degree  to  which  the  user  can 
determine  the  order  in  which  he  performs  different  subtasks. 
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Interfaces  that  give  the  user  extensive  control  of  interactions  require 
greater  resources  for  context  management.  They  generally  require  more  initial 
learning  tine  to  be  used  effectively.  However,  they  can  enable  more  efficient 
use  by  experienced  users.  Interfaces  that  assume  most  of  the  initiative  in 
interactions  require  some  internal  representati on  of  an  acceptable 
organization  for  the  task.  Thus,  they  are  more  difficult  to  implement  for 
complex  tasks.  They  can  make  heavy  use  of  closed  input  modes  and  thus  can 
enable  easy  use  on  simple  tasks  by  naive  users.  The  need  for  users  to  take 
initiative  increases  as  the  scope  of  a  task  increases.  At  the  very  least, 
they  need  th‘  ability  to  return  to  earlier  interactions  to  change  their  inputs 
or  review  the  events  without  losing  the  intervening  interactions.  One  method 
for  dealing  with  the  user  control  issue  is  to  provide  capabilities  for  both 
full  user  controi  and  for  "hand-holding"  by  the  user  interface,  with  the 
ability  to  switch  between  modes  as  desired.  This  approach  requires  maximal 
support  for  context  management  and  more  intelligence  in  controlling  system- 
determined  interactions  than  can  be  provided  by  a  rigid  script. 

USER  KNOWLEDGE  REQUIREMENTS.  Any  system  design  must  consider  what  knowledge 
users  must  have  to  use  a  system  and  how  they  will  obtain  it.  It  is  obvious 
that  the  user  should  not  need  to  learn  more  than  necessary  to  perform  his  task 
in  an  effective  way.  Layered  designs  can  help  minimize  user  knowledge 
requirements,  for  example,  insulating  the  user  interface  functionality  from 
the  underlying  system  structure  helps  assure  that  the  user  need  not  know  about 
the  host  operating  system.  However,  other  knowledge  of  the  system  is 
necessary  to  promote  effective  interactions,  just  as  a  conversation  between 
two  people  requires  certain  shared  knowledge  and  assumptions.  At  the  very 
least,  the  user  must  have  knowledge  about  the  models  of  the  task  domain  that 
the  system  incorporates  so  that  he  can  use  them  or  map  his  own  models,  if  any, 
onto  them;  without  such  knowledge,  the  user  cannot  predict  the  effects  of  his 
i nteractions . 

The  user  can  also  perform  more  effectively  if  he  has  an  understanding  of 
the  interaction  schemas  the  system  employs.  This  can  include  various  types  of 
k  nowledge:  the  plan  underlying  system- control  led  interactions,  the  feedback 
he  can  expect  in  different  situations,  the  syntax  of  open  input  mode 
languages,  and  the  conventions  for  accessing  and  managing  context  information. 
Host  of  these,  of  course,  depend  on  how  other  issues  regarding  the  user 
interface  have  been  resolved,  so  in  effect  the  ocher  issues  all  have 
.  ipl irations  for  user  knowledge  requirements. 

Approaches  to  atisfying  user  knowledge  requirements  may  bo  categorized 
e.  external  or  ir  ial  to  the  target  system.  External  approaches  may  be 
based  on  either  "se  i  ectTon  or  training.  Selection  of  users  whose  aptitudes  and 
I- ior  experience  better  equip  them  to  interact  with  a  system  is  possible  in 
’imit.wj  cases  where  the  system's  purpose  does  not  require  universal 
u.ce  . s i b i I i fy  by  a  population.  Thus,  for  example,  while  it  is  necessary  for 

■  /--ry  t<  Her  in  a  bank  to  access  a  system  fur  recording  transactions  with 

'  ii.  towers,  it  may  only  tie  necessary  for  a  subset  of  them  to  access  a  system 
for  establishing  new  rustoinnr  accounts.  Training  based  on  off-line  materials 
and  presented  by  human  specialists  is  a  conrnon  method  for  satisfying  user 

■  new1 edge  requ ; remen ts .  This  external  training  is  typically  not  adapted  to 
user  needs:  it  is  unnecessarily  expensive  for  some  trainees  and  insufficient 
for  others. 
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Internal  approaches  to  satisfying  user  knowledge  requirements  attempt  to 
increase  accessibility  by  making  presentation  of  the  knowledge  concurrent  with 
use  of  the  system  and  adaptive  to  user  needs.  These  approaches  combine 
several  facilities  in  the  system's  user  interface.  Usually,  default  control 
of  interactions  is  assigned  to  the  system;  that  is,  the  system  does  maximal 
"hand-holding"  for  new  users.  The  interface  also  may  provide  integrated  on¬ 
line  documentation  and  assistance.  In  any  situation,  the  user  can  follow  a 
simple  protocol  to  obtain  a  description  of  what  is  happening  and  what  is 
expected,  without  destroying  context  of  the  interaction.  When  the  user  makes 
an  entry  error,  corrective  feedback  and  tutorial  descriptions  are  presented  or 
available.  In  effect,  indi vidual ized  training  is  delivered  as  needed  and  on 
request  while  the  user  is  using  the  system.  Another  feature  seen  in  internal 
approaches  is  layered  capabilities.  Instead  of  opening  the  full  functionality 
of  the  system  to  the  user,  complex  functional i ty  is  made  accessible  over  time 
either  under  the  control  of  the  user  or  an  external  monitor.  Thus,  users  need 
not  satisfy  all  knowledge  requirements  before  they  can  effectively  use  a 
system  for  some  subset  of  activities. 

DISCUSSION.  The  major  conflict  in  resolving  most  issues  about  user  interface 
design  is  in  balancing  the  desire  for  access  by  new  and  casual  users  against 
the  desire  for  efficient  use  by  knowledgeable,  long-term  users.  These 
different  classes  of  users  are  generally  served  best  by  different  input  modes 
(closed  vs.  open),  balance  of  system  vs.  user  initiative,  and  degree  of 
ontrol  over  preservation  and  access  to  context.  The  difficulty  in  addressing 
both  sets  of  needs  in  a  single  system  lies  primarily  in  the  cost  of 
engineering  an  array  of  alternative  mechanisms  and  the  difficulty  of  designing 
techniques  for  smooth  transition  among  them.  Engineering  costs  include  both 
the  software  design  and  the  hardware  resources.  Increasing  the  features  of  an 
interface  lowers  the  quality  of  Its  responsiveness,  which  can  make  a  system 
unacceptable  to  both  classes  of  users.  Maintaining  responsiveness  as  the 
number  of  features  increases  requires  enhancing  the  capacity  and  speed  of  the 
hardware  system.  If  hardware  resources  are  fixed,  then  the  interface  design 
must  involve  tradeoffs  in  functionality. 

INPUTS  FROM  SYSTEM  BUILOERS  AND  POTENTIAL  USERS 

As  required  in  the  contract  statement  of  work,  we  interviewed  several 
potential  DE-users  of  an  IK AS  in  order  to  gather  data — expressions  of  their 
"needs  and  expectations"  —  that  might  be  relevant  to  resolving  issues  in  design 
of  the  system's  user  interface.  We  also  obtained  from  the  potential  users 
information  that  bears  on  the  question  of  whether  they  can  articulate  the 
knowledge  about:  their  task  domain  that  the  Alternative  1  IKAS  would  bo 
des  i  gnod  t.o  •>  1  ici  t . 

In  addition  to  potential  users,  we  interacted  with  system  builders  who 
have  worked  with  DF.s  to  build  training  systems.  We  discussed  their  opinions 
about  the  feasibility  of  constructing  an  IKAS  that  could  interact  directly 
with  DE-users.  We  also  interacted  w’th  system  builders  with  experience  in 
building  expert  systems  for  knowledge  acquisition.  The  following  sections 
report  the  results  of  this  data  collection  effort. 
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Selection  of  respondents.  We  sought  Navy  DEs  who  had  had  some  experience  in 
deveTopnent  or  "use"  "of"  training  technology.  We  wished  to  establish  with  our 
interviewees  a  mutual  understanding  and  rapport  concerning  our  objectives. 
Therefore,  we  decided  that  communication  would  be  enhanced  by  (a)  our 
knowledge  of  the  respondent's  field  of  expertise,  and  (b)  respondent's 
familiarity  with  computers  and  computer-based  instruction.  Practical 
limitations  included  our  ability  to  identify  and  arrange  access  to  active  Navy 
personnel  with  demanding  work  schedules. 

Our  OF  sample  comprised  four  individuals  contacted  either  through  our 
colleagues  in  the  Navy  RAD  community  or  through  our  activities  in  other  Navy- 
sponsored  RAO  projects.  Three  a^e  DFs  in  surface  warfare  tactics,  and  one  is 
a  OF  in  sonar  operation  and  interpretation.  These  individuals  were  assigned 
to  positions  at  the  Navy  Personnel  RAO  Center  ( NPRDC )  and  the  Fleet  Combat 
Training  Center  (FCTC)  in  San  Oiego,  California,  and  had  authority  to  interact 
with  Navy  RAD  contractors  at  their  own  discretion  as  part  of  their  job.  They 
were  either  actively  involved  in  training  technology  development  or  in 
training. 

One  group  of  system  builders  comprised  six  civilian  Navy  employees  at 
NPRDC.  These  individuals  are  engaged  in  RAD  efforts  to  design  and  implement 
prototype  training  systems  for  a  variety  of  applications.  Some  of  these 
individuals  had  been  involved  in  the  same  system  building  efforts  in  which  our 
OF  respondents  had  participated. 

We  also  interacted  with  two  system  builders  at  FCTC.  These  individuals 
were  Navy  officers  who  were  building  a  system  to  be  used  in  constructing 
embedded  training  for  the  Navy  Tactical  Data  System  (NTDS)--in  some  sense,  an 
IK AS .  Their  particular  concern  was  in  user  interface  design  to  support  access 
to  the  system  by  untrained,  low-skilled  users  (i.e.,  not  DEs)  whose  task  would 
be  to  enter  existing  hardcopy  specifications  of  curricula  into  the  system. 

We  interviewed  one  other  Navy  system  builder  at  the  Naval  Oceans  System 
Command  (N0SC5  in  San  Diego.  This  individual  had  been  identified  by  our 
respondents  at  NPRDC  and  NOSC  as  being  currently  involved  in  building  a 
knowledge  acquisition  system  for  a  Navy  expert  system. 

At  various  points  during  the  project,  we  discussed  with  colleagues  from 
the  Stanford  I'irversity  AI  cotmunity  the  feasibility  of  automating  knowledge 
acquisition  from  OF s  and  our  particular  concepts  for  an  IK AS.  These 
individuals  all  have  experience  as  knowledge  engineers. 

Interview  method.  Interviews  were  arranged  and  conducted  by  Keith  Wescourt  o'. 
the  project  staff.  Prior  to  the  scheduled  interviews,  copies  of  the  project's 
objectives  from  the  contract  Statement  of  Work  were  mailed  to  the  DEs.  Each 
confirmed  that  had  read  this  material  prior  to  the  interviews. 

The  interviews  were  held  at  the  respondents  work  places  over  a  2-day 
period,  fly  their  own  choice  and  our  agreement,  the  DFs  at  NPRDC  and  at  FCTC 
were  interviewed  in  single  group  meetings.  The  FCTC  meeting  also  included  the 
two  Navy  officers  who  were  developing  the  NTDS  curriculum  development  system. 

The  DF  interviews  were  conducted  in  an  informal  style.  Our  staff  member 
introduced  topics  from  a  prepared  agenda  when  and  If  they  seemed  appropriate 
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based  on  the  respondents'  apparent  knowledge  and  opinions  expressed  during  tne 
interview.  Since  the  topics  were  for  the  most  part  outside  the  day-to-day 
concerns  of  the  DEs,  they  were  encouraged  to  respond  to  inquiries  based  on 
their  past,  perhaps  idiosyncratic,  experiences.  For  example,  they  were 
encouraged  to  evaluate  user  interface  input  modes  by  reference  to  particular 
computer  systems  they  had  used.  The  discussion  with  each  group  lasted  2  to  3 
hours.  Both  were  tape  recorded  with  the  permission  of  the  participants. 

The  interviews  with  Navy  system  builders  were  conducted  as  informal 
collegial  d1' scussions.  The  1-  to  2-hour  interviews  at  NPRDC  were  also 
recorded.  Interviews  with  other  Navy  system  builders  were  arranged 
opportunistically  during  the  visit  to  NPRDC.  These  interviews  were  shorter 
(10  to  20  minutes)  and  were  not  tape  recorded. 

Discussions  with  artificial  intelligence  researchers  with  expert  systems 
building  experience  in  the  civilian  world  occurred  informally  at  several  times 
in  the  project.  We  will  not  describe  these  discussions  in  detail,  but  we  will 
indicate  when  the  opinions  expressed  during  these  discussions  either  reinforce 
or  contradict  those  obtained  from  the  Navy  system  builders. 

Interview  agendas .  The  agenda  for  discussions  with  the  Navy  system  builders 
incTuded”tTie ToTTowing  topics: 


a.  What  is  the  most  difficult  aspect  of  knowledge  engineering? 

b.  How  feasible  is  automation  for  knowledge  acquisition?  To  what 
extent  does  feasibility  depend  on  already  having  task  and  domain  specific 
Knowledge  embodied  in  the  automated  knowledge  acquisition  system? 

c.  What  user  interface  features  does  a  computer  system  require  to 
support  direct  use  by  Navy  DEs? 

These  topics  were  discussed  in  the  context  of  an  initial  description  of  our 
concept  for  an  I K AS  generic  to  a  class  of  tasks.  The  respondents  introduced 
other  topics  which  they  thought  were  relevant  to  our  system  concept,  including 
the  possibility  of  applying  the  concept  to  the  enhancement  of  an  existing 
instructional  system  prototype  (Alternative  3,  described  in  Section  IV). 

The  agenda  for  the  discussions  with  the  DEs  included  a  variety  of 
specific  questions  for  each  of  the  following  general  topics: 


a.  What  is  your  experience  in  using  computer  systems?  (How  long? 
lor  what  purpose?  What  physical  systems?  What  conditions  of 
accessibility?  What  training?) 

b.  What  makes  a  computer  system  "friendly"  to  its  users7  (What  are 
the  strengths  and  weaknesses  of  systems  you've  used?  What  general 
problems  do  you  perceive?  What  changes  would  remedy  them?  What 
preparation  should  be  required  before  first  using  a  system7) 

c.  How  in  you  see  the  Navy  with  respect  to  the  general  trend  for 
increasing  computerization  of  society?  (How  do  you  place  yourself  with 
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respect  to  that  trend?  With  respect  to  the  rest  of  the  Navy?  Are  there 
important  differences  within  the  Navy  by  age  or  job  area?) 

d.  How  useful  have  computers  been  for  operational  and  training 
functions  for  your  job  area?  (Arc  there  other  opportunities?  What 
inputs  from  DEs  like  yourself  would  he  needed  to  develop  the 
opportuni ti os  ef fecti vely? ) 

o.  What  experience  have  you  had  in  interacting  with  groups 
developing  sj  stems  for  operations  or  training?  (How  did  the  involvement 
occur?  Were  there  specific  rewarding  and  dissatisfying  aspects?  How 
much  background  knowledge  had  to  be  exchanged  before  you  felt  progress 
occurred?  Do  you  have  any  thoughts  on  how  computers  might  have  aided  the 
system  development  process?  What  are  the  advantages  and  disadvantages  of 
becoming  involved  in  such  projects?) 

f.  What  is  the  nature  of  knowledge  in  your  field  of  expertise? 

(How  much  of  what  is  required  for  performance  can  be  learned  from  formal 
coursework?  How  important  are  exercise  and  OJT?  Is  it  hard  to  describe 
the  knowledge  derived  outside  formal  coursework?  How  do  you  judge  your 
ability  to  evaluate  another's  performance  in  your  job  area?  When  you  see 
someone  make  a  performance  error  in  your  job  area,  is  U  easy  to  see  its 
causes?  How  long  would  it  take  to  describe  your  knowledge  of  how  to  do 
your  job1  How  would  you  help  yourself  remember  and  organize  the 
knowledge  you  would  describe?  Do  you  think  there  are  computer  techniques 
that  could  allow  you  to  communicate  your  description  directly  to  a 
sys tern1) 

Time  constraints  and  differences  in  the  DE ' s  backgrounds  made  it 
impossible  or  inappropriate  to  discuss  with  each  all  of  these  topics.  For 
those  topics  covered,  the  DEs  were  asked  for  their  belief  about  how  typical 
their  responses  might  be  of  others  in  their  specialty  and  of  how  they  might 
expect  responses  from  individuals  in  other  specialties  to  differ.  We  adopted 
this  tactic  in  recognition  that  although  our  approach  to  selecting  and 
interviewing  DEs  could  obtain  candid  and  detailed  data,  these  data  might  bo  o 
limited  generality.  By  asking  explicitly  about  generality  we  hoped  to  obtain 
some  qualitative  characterization  that  would  be  of  value  for  preliminary 
system  design  of  an  IKAS  intended  for  a  broader  user  population  or  other  tad 
domains.  In  any  case,  the  current  posting  of  these  DEs  to  R&D  and  training 
billets  suggests  that  they  are  representati ve  of  the  individuals  who  would  be 
prime  candidates  for  the  role  of  user  of  an  JKAS.  Therefore,  it  is  reasonabl 
to  assume  ‘'hat  the  DES'  remarks  represent  the  expected  norm  for  other, 
potential  users  in  their  job  specialty,  although  the  generality  of  their 
remarks  to  other  task  domains  and  domain  experts  may  be  more  questionable. 

Pi:?  HVIrrf  Rf  ‘  !>l  id:  SYSTEM  BUILDERS. 

On  the  feasibility  of  automating  knowledge  acqui sit. ion.  Every  system  tn.ilder 
wn  '.poke  with  agreed- with "'oiTr  analysis"  tnat  repTac f ng  the  IT  with  a  generic 
knowledge  at  quisition  system  is  far  beyond  the  state-of-the-art.  Their 
masons  included  those  we  have  stated  previously. 
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a.  The  extensive  knowledge  and  interpretive  skill  required 
initially  to  communicate  with  DEs  and  to  organize  interactions 

<  >  f fee t i voly 

b.  The  dependence  of  knowledge  acquisition  on  representation 
formalisms  used  in  the  host  system 

■  .  The  dependence  of  choice  of  formalisms  on  domain  features  and  on 
specific  capabilities  to  be  realized  by  the  knowledge-based  system, 
nei  tiler  of  which  is  known  initially. 

The  specific  features  of  the  interaction  between  the  KE  and  DE  that  were 
perceived  as  difficult  to  automate  included: 


a.  The  KE '  s  role  as  ”filte>'or  and  ;-ucturer",  not  just  a  "bridge" 
for  •>  rnnsforr  ing  t iie  DEs  descriptions 

b.  Determination  of  tne  consensus  within  the  do  min  for  a  specific 
It  '  s  knowledge 

o.  The  KE 1 s  use  of  evolving  semantic  and  pragmatic  knowledge  to 
control  the  topics  and  granularity  of  interactions . 

The  last  of  these  features  bears  on  the  issues  of  system  initiative  and 
flexibility  in  user  interface  design.  It  questions  whether  a  system  that 
takes  active  initiative  in  a  knowledge  acquisition  system  can  ask  the  "right" 
questi on$--those  effective  for  obtaining  knowledge  while  motivating  the  DE-- 
using  dialog  management  rules  based  only  on  the  syntax  of  knowledge 
formal i sms. 

There  were  v.rying  degrees  of  concurrence  that  an  automated  knowledge 
acquisition  system  generic  to  a  class  of  tasks  is  feasible  at  this  time.  This 
concent  was  perceived  as  embodying  a  moderate  level  of  risk,  but  one 
appropriate  ;  r  a  research  effort.  That  is,  even  if  the  effort  failed  the 
process  would  have  provided  valuable  knowledge  to  the  community  of  system 
builders.  One  of  the  respondents  felt  that  the  major  problem  would  involve 
operationalizing  the  notion  of  a  class  of  tasks  for  use  by  an  automated 
knowledge  w  riisifion  system.  He  wondered  whether  the  requisite  class-generic 
knowledge  iijuld  be  derived  from  one  exemplar  of  the  class  or,  if  not,  how  many 
exemplars  -lust  be  studied.  Another  concern  involved  the  uncertain  difficulty 
if  usin')  thu •;  knowledge  as  a  KT  does  in  dialogue  management.  Respondents  who 
had  developed  a  "generic"  instructional  system  (CMS--see  Alternative  3, 

Section  IV)  believed  that  that  system  instantiated  the  concept  of  a  class  of 
tasks.  They  believed  that  the  difficult  problem  in  implementing  a  knowledge 
inquisition  system  for  such  a  system  lies  mainly  in  creating  a  user  interface 
accessible  to  DFs. 

On  DF.  knowledge.  Although  it  was  no*'  included  among  topics  to  be  discussed 
with  the"  system  builders,  the  question  of  the  nature  of  DEs1  knowledge  and  its 
implications  for  our  IKAS  concept  emerged  as  a  serious  issue.  One  aspect  of 
that  i 3 sue  is  the  lack  of  consensus  in  expert  task  knowledge  across  DEs.  The 
respondents  at  NPRDC  who  had  work  i  closely  with  Navy  DEs  to  build  prototype 
training  systems  for  several  tasks  (TAO  threat  classification,  EW  signal 
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classi  ficat  i  on,  r .  .1  v  i  \  on )  emphasi  ;:,-o  that  iif ■.  v^ry  s  •  uni  t  icar.tly  if-  their 
k fiow  1  oil  ,0  iTu!  hi -;n -1  <*v"i  approach  to  their  tasks  /’  tuon  -.1 1 1  fc-renees  '.an  r<i  ,f 
even  in  tne  .  i  h  ■ r-  >  r  ■; :  «  »ri  .it  ioni  in  t.;:»  1 0  v  •?  1  of  task  (-i'i  !  ormancp  ;  that  1 

two  t-'.port-  m.  p  •  f . . ;  :  i  v  •  l  w  ■  •  h  di  f  for*.-  >t  kncwle"' Mid  :  *  . .  *  > p i •  •  ■  ch<  • 

to  tho  tas1  .  ’these  vari.it  ions  rer'tp  the  lack  of  *  t-r,.w  "expert  model  “  and 
of  good  c ’  t •' ■  ::  for  ;.k  -fonrie.nrr  ova  \iut '  ; Ur’  those  ''-?vy  tasks.  Too-  system 
builders  m  010  I  on  •'*  ’  know!  ’re;  mm  •  v;  r :  ;  1  r  :  0  •  reeir-  r. 

cons  i  derail  1 t :  op  .-  tti  fits,  c-c..  i oration,  tod  i.inov.itia  *o  define  a 
C  omoe  tone;.  *  for  .1  r  a  s  k  -  ~u  n  1  1  k  s  see  k  mow  '  e  d.ge  or  g  1  r-^e  ri  ng  rouu  rot)  for  t  fit  ’ 

well-knowr  ■  ■■;  or;  so  ’  tor  •  v- ■:  ems .  One  res  pendent  suggest;  d  that 
concensus  r.o.-  '  ;  ■  .1  :i  1  ■  r  ..  -  .•  or  "o.'icter  *  vuc:-  o  "team 

propul  si  on  or  r«t>!  sr  oper*  t  .or’  ;  nis  v.o'iiiiitoit  was  bused  on  knowledge  of  on 
•ji'KOf  project  ; ...  Sic;’  ;,  a  w. edge-based  training  system  f  0  c teats 
propu'  si  on  plant  ■  i.t  •  itfor. . 

'hr  1 1 1  ■  ■  ,  1  ■;  •  whi.  worked  wit!1  f»;  s  tl  s;,  indicated  *  •  r<-w, 

i  ■  :nv(  . ) 1  ■>  ,■  ..»u  i '■  -  r'.’'**'  11  ■  ef  fer.  M  v<-  lusrru  t . ‘in  of  their 

1  j  1 1  y  .  >'t  .1  t.  .■  ■'  ;  •  ;  :  ■  'M'.n  if,  ' )  :  s  f;sv<  f  or  go  t ;  ■  *n  wfjt  ip  ■  s 

; :  k,.  to  he  ,t  1, •  -r  .  ”  n  ,  1  ui  .m  h  ai  -  *h"  if  -m  IKA>.  (rot. ..type 

c,y  s  T  e"! .  h.isf-  ;  <>■•.  v . ,  i  ■  \  / 1  ■-  s:  1  a,v>  e’  f."^  a,s  .  *;  ,s '  i .  m  froii  s  have  been 

i  fie  f  f  et  f.  i  Vi  •  -n.,1.  ,■  a  t;ve  t>-  ■•■.■c  s  who  hive  .st  ’  fhe'1'..  eirni  these 

sy'sfe  is  w? *  e  .’leni  ;:;■>•••  :  j  syste-  's  1  i  i  1  cer;  wi  tn  v  w  .  ■'  -u  r^pet  \  so,  they  wt-re 

oval  j.»t«  i  nit;  r'f  i  rift  i'V;,  in  acltn  t  ion  ,  the-  ;f.s  iiekn.  w  i  ••  .geo  that  the  training 

methods  ,ve;  <•  s.;p-rior  those  th  :>  had  jiroposod.  'v'O'-  -espondents  offered 

an  opinion  tbs  ■  tr.*  j**s  ni  r,,-.;  1 1  y  tne  Al  tern.it  »v>';  i  1  iincept  of  an  iCAa  for 

performance  mo  ;  '  - -ssri  of  the  types  and  causes  of  i  ra ’ tier*  errors .  f  hey 

suggested  that  w.v’  "e  hr  ,  ire  able  to  descri be  t>  t  -es  of  errors  a  trainee 
■u'ght  co’Tc't  wi-Chi  ;•  -r  f.  m  nu  ,>  ocedure ,  the.  j...  :-ct  lake  ";:.t  Knowledge 

about  the  onderl  v '  ng  ; uses  of  toe  err:- .  ( 1  .e. ,  ths  trainees'  "conceptual 

bugs")  or  abf;  r  Vis  ■■.••urtiona  i  interventions  that  are  effective  .or  eliminating 
errors . 

On  unking  svs'i  ir •■.  i h ! »•  to.  .  Tne  sv.  cr's  >a  i  J»  per,. et  ve  a  range  >' 

sk  i  IT  s  in  <inri  ,•  v '  fijo  s  to  ware  compo  tor  u  ’■  :;v  hi- s.  •  •  •'i  the  .  r  ;  0  ■  r.t  Of  v'c-w, 

the  existence  et  ;  few  fit's  wi  thi  n  ?  spec  i  a  I  ty  wi’c  siv*:-1  covpwt  i  n  ■  rack  grifcnu 
would  he  ui.f';  te  iiistify  1  svstro,  with  wfpe  ",  r  cobd  interact 

.1  i  1 1  y  .  hi". 1  •  e.%  -  i ;  .-•  i  f  ■_. -1  .  ;rr-  ‘.’re  ;•  ••*•!.  .■  •’■  *  •  1  rt  i  <  'Cifr 

otn  rvn  tly  in  "v  '  ■’  ..C'.:  ~  ■■  ■ ‘  ■  •  >\  '  ’  • -h  '  oy 

nrhilruil,’  •,:■!.■  witrtd  1  l  .i .  ! wc.  •  .  f  '  ■  -  Man./i1 

k now1  fM  ]•'  *’n  \:  .\t  •*»*  °  t.  wi  •  ii  n  1 r  »  M  )  lnv',‘ 

J  .<»#  1  hfy!.  ^  J  i  ‘  ■]  k' t  *  i  '  .  ■  '  f.vi  ■  f  *■  P  it  ••  *  ’  1  mV 

autoi’i-.'f.iijn  .•mu'.:  :>c  ....  •  • 1  < l  . 

r he  ■  •): •'■)'!■•;■. v ii  1  i  a  nij'shf-r  of  opinions  about  •  otcrfarc  design.  •  hose 

wry  based  fh'  1  •  efiurss  to  build  systems  fo'"  use  by  !>Ls  on  tasks  such  as 
( 1 1  r  r i c  u 1  urn  dr . 1  in  and  instructional  system  management.  Again  .  the  Of  $ 
participating  1 n  ‘•hese  efforts  had  good  skills  for  non-compu  ..or  professi  o.nal  s , 
were  vol unt.ee-- 1  ■  .i  hence  presumably  motivated.  Thus,  the  respondents* 


As  we 


US’S 


the  Dl.s  agreed  total  1  y  with  this  assessmrnt  of  thei 


i  (Wiser) S II'.  . 


NAVTRAEQUI PCEN  82-C -0151-1 


opinions  are  based  on  a  "best-case"  appraisal  of  their  experience.  However, 
they  also  indicated  how  they  expect  the  interface  would  have  to  differ  for 
less  experienced,  but  equally  motivated,  users  of  a  system.  Their  suggestions 
are  consonant  with  the  conventional  wisdom  about  interface  design.  In 
addition,  a  few  specific  suggestions  onented  to  the  I K AS  concept  were 
proposed . 

The  inputs  concerning  interface  design  include  those  offered  by  the  two 
officers  at  t-'CT"  who  were  developing  an  operational  curriculum  entry  system 
for  NTDS  trii.'ing  materials.  ’ hei r  users  were  untrained  enlisted  personnel 
and  their  ideas  about  interfaces  were  oriented  toward  state-of-the-art 
techniques  for  supporting  such  users. 

The  respondents  at  MPRDC  have  found  that  the  skilled  and  motivated  DEs 
who  they  have  worked  with  have  learned  without  difficulty  to  access  systems 
through  the  same  user  interfaces  used  by  the  system  builders  themselves.  This 
includes  learning  operating  system  commands  to  invoke  the  application 
programs,  interacting  with  application  programs  that  require  user  initiative, 
and  using  poorly  documented  and  inflexible  programs  (typical  characteri sti cs 
of  incomplete  research  prototypes).  One  comment  about  the  experience  of  a  HE 
using  CMS  is  revealing,  however.  The  DE  learned  to  use-  the  system,  but  had 
difficulty  in  accomplishing  his  objective  of  constructing  a  data  base  for  a 
new  application  of  the  system.  This  emphasizes  the  distinction  between  how 
the  mechanical  aspects  of  an  interface  influence  accessibility  and  how  they 
influence  the  effectiveness  of  access.  In  the  latter  case,  the  user  interface 
can  be  more  or  less  effective  at  supporting  the  user's  conceptual  problem 
solving.  Similarly,  civilian  system  builders  also  indicated  that  since 
encoding  knowledge  into  a  system  is  difficult  even  for  a  KE,  a  major  problem 
for  automated  knowledge  acquisition  is  providing  conceptual  support,  not  the 
mechanics  of  system  access. 

Several  comments  were  made  about  features  .,f  the  user  interface  design 
that  would  in  general  he  associated  with  the  implementation  of  conceptual 
>ti;>:»or t .  One  respondent  suggested  that  HE s  are  both  m.-.re  effective  and 
com fortubl e  with  the  elicitation  of  procedural  knowledge  while  solving  actual 
problems.  Another  respondent  suggested  two  approaches.  One  was  to  support 
specification  of  new  knowledge  (new  inputs)  by  indicating  how  it  is  different 
from  existing,  analogous  knowledge,  instead  of  oy  composition  of  primitives 
from  scratch.  The  second  was  to  support  completion  of  a  knowledge  base  by 
generating  structural  (syntactic)  variations  automatical ly  for  the  user  to 
filter,  instead  of  by  relying  on  the  user  to  remember  or  formulate  the 
variations  on  his  own  initiative.  The  respondents  did  not  identify  these 
features  with  the  need  for  any  particular  characteri sties  of  the  physical 
interface. 

With  respect  to  the  question  of  user  interface  requirements  for  DE  s  with 
less  skill,  experience,  and  motivation,  the  respondents'  replies  were  quite 
loner. il .  One  of  then  believed  that  fer  such  user,,  the  interface  should  take 
maximal  initiative.  She  held  that  t^e  need  tor  structure  outweighed  fhe 
possible  adverse  implications  for  flexibility,  even  11  the  resulting 
interaction  was  so  rigid  and  pedantic  as  to  offend  sonic  users.  Another 
respondent  pointed  out  that  interfaces  requiring  typing  severely  reduce  the 
c-f fecti veness  for  users  with  less  experience.  His  comment  seemed  to  suggest 
that  linguistic  organization  of  interaction,  as  well  as  the  input  mode  and 
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medium,  was  less  effective  than  the  alternatives  tor  such  users. 

The  PC  Ft  respondents  reported  positive  results  for  a  system  (LTRAN, 
"Lesson  TRAMslator")  used  by  inexperienced  users.  The  features  of  the  FTP'11.' 
interface  include  maximal  use  of  menus  responded  to  by  function  keys,  sealin' 
visual  organisation  of  the  interaction,  a  pointing  device  for  graphics 
interaction,  "intelligent"  on-line  "help"  facilities,  and  a  user -invoked 
"undo"  function  for  correcting  mistakes.  The  spatial -visual  organization  is 
enabled  by  the  nature  of  the  task--  the  construction  of  training  lessons  for 
delivery  on  the  MTOS  graphic  display.  The  user  of  I.  TRAM  continually  sees  the 
lesson  as  it  will  appear.  The  FCTC  respondents  feel  strongly  that  the  user 
interface  features  in  LTRAN  are  important  for  supporting  inexperienced  users 
of  other  systems  as  well.  In  particular,  the  use  of  graphics  displays  and 
interactions  increases  accessibility.  However,  they  also  indicated  that 
spatial-visual  organization  would  probably  be  less  effective  for  tasks  in 
which  there  is  no  existing,  natural  use  of  graphics.  They  believed  that 
requirements  to  learn  an  artificial  visual  model  could  increase  the  difficulty 
of  using  a  system. 

To  summarize  our  discussions  with  system  builders,  we  found: 


a.  They  believe  that,  any  effort  to  automate  knowledge  acquisition 
must  limit  the  intended  generality  of  the  system. 

b.  They  emphasize  the  difficulties  in  providing  conceptual  support 
as  the  major  problem  in  implementing  a  limited  knowledge  acquisition 
system  for  direct  use  by  DFs. 

c.  They  deemphasize  the  importance  of  the  physical  features  of  the 
interface  to  be  used  by  DFs,  because  they  believe  that  in  general  there 
are  some  DEs  in  any  specialty  with  the  skills  and  experience  to  use  any 
functionally  complete  interface. 

INTER V/F  W  RESULT:  DOMAIN  EXPERTS. 

On  computing  background  and  att’cudes.  All  of  the  OF  respondents  had  a 
work  Trig  knowledge  of  interactive  computer  systems,  a  real  is  Me  attitude  ’bout 
the  oapalvl  itm*s  and  uses  of  computer--,,  and  a  '"unrjblo  attitude  about  the 
introduction  of  n-‘w  computer  technology  into  training  and  operational 
environment for  their  specialty.  One  of  the  LCTi  respondents  had  had  ta-ui' 
coursewon  in  i  ’Xnput.ers  at  the  Naval  Academy  and  in  an  MBA  program  at  a 
civilian  university.  The  work  at  the  Naval  Academy  included  BASIC  programming 
and  use  of  a  LAB  system  in  a  ship  design  class.  He  had  not.  done  any 
programming  since  leaving  the  Academy  some  4  years  earlier,  however.  Since 
that  time,  he  had  used  i  variety  of  embedded  computer  systems  within  the  NTI3S 
and  other  tactical  warfare  systems.  At  FCTC,  he  had  a  major  responsibility 
for  using  NAVI  Ah,,  a  microcomputer-based  training  system,  in  a  supervisory 
roll',  lie  defined  new  exorcise  scenarios,  played  the  role  of  game  director  in 
exercises,  and  managed  trainee  data  in  the  system. 

The  other,  more  senior,  FCTC  respondent  had  no  formal  exposure  to 
computing.  However,  his  position  required  him  to  use  various  computer-based 
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training  and  administrative  systems  at  FCTC.  He  also  had  experience  using 
personal  computer  systems  outside  his  work. 

The  NPRDC  respondents  had  considerable  computer  skill.  One  had  an 
extensive  formal  background  from  work  in  Masters  and  Ph.D.  programs  at  the 
Naval  Postgraduate  School,  which  included  4  years  cf  programming 
simulations  in  FORTRAN.  At  NPROC,  he  was  the  DE  member  of  a  team  developing 
tactics  training  systems.  Part  of  that  work  included  an  independent  research 
effort  in  which  he  was  developing  on  a  micro-computer  in  PASCAL  an 
interactive,  graphics-based,  training  simulator.  In  addition,  he  accessed 
NPRDC's  VAX/UNIX  timesharing  system  on  a  daily  basis  for  computer  mail  and 
text-processing  activities. 

The  second  NPROC  respondent  was  a  senior  non-commissioned  officer  with 
self-trained  computing  skills.  He  had  taken  a  COBOL  programing  course  and 
had  taught  himself  PASCAL  programming  at  NPROC.  He  reported  that  he  was 
motivated  to  find  ways  to  develop  and  use  his  skills  in  his  job  at  NPRDC,  even 
though  this  was  not  strictly  required.  He  regularly  used  computers  to 
interact  with  progranmers  implementing  systems  in  groups  in  which  he 
participated.  He  also  was  the  DE  who  had  worked  on  adding  a  new  database  for 
his  specialty  (sonar  interpretation)  to  an  existing  training  system  prototype 
(CMS).  Thus,  he  had  actually  been  engaged  in  an  "automated  knowledge 
acquisition"  effort. 

With  regard  to  the  generality  of  experience  and  attitudes  we  encountered 
in  our  sample  of  OEs,  the  respondents  at  NPROC  recognized  their  relative  depth 
of  experience.  The  tactics  expert  believed  his  range  of  experience, 
particularly  in  programming,  is  uncommon  for  other  officers  in  his  specialty 
and  exceeded  only  by  officers  whose  specialty  is  computer  systems.  He  noted 
that  the  other  DE's  knowledge  and  motivation  was  particularly  rare  for  non¬ 
commissioned  officers  with  real  expertise  in  some  operational  specialty.  That 
DE  concurred  by  pointing  out  that  other  noncoms  at  NPRDC,  including  those  in 
his  specialty,  had  a  very  different  approach  to  their  involvement  in  projects 
developing  computer-based  systems. 

The  OEs  at  FCTC  see  their  experience  and  attitudes  as  more  common,  though 
far  from  universal.  They  state-!  that  among  Navy  officers  in  all  specialties, 
one  can  expect  a  functional  computer  background  from  all  who  attended  the 
Naval  Academy,  some  of  those  who  attended  the  Naval  Postgraduate  School,  many 
who  take  courses  at  colleges  and  universities  on  their  own  initiative,  and 
those  whose  specialty  includes  operational  equipment  that  contain  computers. 
With  regard  to  the  last  category,  one  of  the  respondents  noted  however  that 
the  functionality  of  most  embedded  systems  is  less  complex  than  that  of  the 
typical  stand-alone  word  processor.  One  of  their  coments  was  that  FCTC  may 
be  somewhat  unusual  in  the  extent  of  its  use  of  computers.  This  may 
contribute  to  initiative  of  Navy  personnel  in  developing  computer  skills. 

They  were  unsure,  for  example,  whether  the  widespread  ownership  and  use  of 
home  computers-- five  or  six  of  the  staff  in  the  TAO  training  group  are  in  this 
category-- is  typu;dl  of  other  Navy  personnel .  One  factor  they  cited  that 
affects  such  initiative  is  the  amount  of  time  required  by  assigned  duties.  It. 
seems  reasonable  that  the  opportunities  and  motivation  for  self-improvement  of 
computer  skills  generalize  to  other  Navy  training  and  development  centers 
where  OEs  come  into  contact  frequently  with  internal  and  contractor  computer 
projects.  OEs  assigned  to  sea  duties  and  to  shore-based  operations  facilities 
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are  less  likely  t.o  have  t'Oth  the  time  and  the  stimulation  to  extend  their 
experience  with  computers,  in  any  case,  the  FCTC  iJF s  believed  it  would  he 
rare  to  find  a  current  officer  who  would  resist  learning  to  use  a  new  system 
if  he  were  told  its  advantages.  They  suggested  that  resistance  to  computer 
technology  occurs  only  at  senior  management  levels.  In  attitudes  toward 
computers  then,  the  Navy  appears  to  he  at  least  as,  if  not  more,  progressive 
than  other  organizations. 

On  tiie  features  of  use>-  interfaces.  The  DEs  ha  I  a  variety  of  opinions  about 
User 'In ter fa  es.  Many  of  Those"  wore  vague  and  general;  others  were  specific 
and  described  in  terms  of  particular  systems  they  had  used.  As  we  noted,  we 
used  our  perceptions  or  these  opinions  to  classify  the  set  of  "issues"  about 
interfaces  we  described  earl k  r  in  this  section.  Somewhat  surprisingly,  the 
DEs  had  more  sophisticated,  detailed  ne^spec ti ves  on  the  physical  features  of 
user  interfaces  than  the  system  builders  we  interviewed.  Some  of  their 
comnents  at  least  uart.  ially  contradict  those  given  by  system  builders.  One 
problem  the  DEs  consistently  nan  during  our  discussion  was  keeping  separate 
the  criteria  they  would  apply  to  systems  used  by  others  (e.g. ,  trainees  using 
-aining  systems)  'ron  those  they  would  apply  to  a  system  they  would  use 
ie.g.,  the  instructor's  fit  terrace  to  a  training  system  or  the  interface  to  an 
IKAS) .  Our  discussion  will  he  limited  to  presenting  comments  that  bear  on  the 
latter  type  of  user. 

Much  of  the  discus  si  or.  with  the  FCTf  respondents  addressed  the  NAVTAG 
system's  interface,  with  wnich  both  they  and  the  interviewer  were  familiar. 
NAVTAG  makes  heavy  use  of  sequential  and  nested  menus.  The  respondents  noted 
that  while  this  feature  was  useful  very  early  in  one's  use  of  the  system,  it 
quickly  became  tedious  for  both  trainees  and  1 nstruc tors.  One  aspect  of  the 
problem  was  responsi veness  the  hardware/software  was  slow  in  displaying  each 
new  menu.  Comments  about  other  systems  odicated  that  responsiveness  was  a 
major  issue  fo r  one  of  the  ofs.  He  suggested  that  while  some  users  could 
tolerate  poor  responsiveness,  other  users  would  elect  not  to  accept  and  use  a 
system  with  poor  response  time. 

A  second  :*mi  with  *i<WT  AC's  use  of  menus  was  the  inability  to 
immediately  to.*  .<  particular  act’ on  when  they  knew  exactly  what  they  wanted 
to  do.  That  is,  toe  system  is  inflexible.  They  -aid  they  would  prefer  a 
command  language  i  r  t.cr  r0ce ,  with  some  minima '■  prompting,  that  would  give  them 
more  initiative.  ;<  addition,  since  they  believe  that  DEs  have  minimal  typing 
skills,  the  i.nmi  nut  language  should  he  terse  and  the  interface  should  be 
flexible  is  its  response  to  spelling  or  format  errors.  One  respondent 
suggested  that  the  menu-based,  scripted  elicitation  of  responses  under  system 
control  was  n  it  objectionable  if  the  task  was  to  enter  a  number  of  connected 
inputs  {as  in  filling  out  a  template),  but  that  the  user  needed  more  control 
when  choosing  the  topic  of  interaction  arid  when  modifying  a  single  feature 
specified  in  a  prior  interaction. 

Doth  !)•  s  agree  !  >  nut  a  rich,  quasi  -  rngl  i  sh  c  norland  language  was 


^Ihe  next  version  of  'JAVTAn,  under  development  on  newer,  more  powerful 
hardware,  could  imerove  regions  f  veness . 
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unnecessary  for  Navy  personnel.  They  routinely  use  arbitrary,  schematic 
information-coding  formats  with  rigid  syntax  in  a  variety  of  communication 
contexts--they  even  have  to  write  computer-parsable  messages.  They  do  not 
object  to  learning  such  languages  and  favor  them  for  their  efficiency  and 
minimal  typing  requirements  over  more  verbose  forms  of  expression. 

Other  responses  about  the  use  of  menus  bear  upon  the  issue  of  context  and 
feedback.  One  comment  addressed  both  the  preservation  and  display  of  context. 
The  DE  criticized  the  fact  that  the  menus  occupied  so  much  of  the  display  area 
and  that  each  menu  erased  the  preceding  interaction.  He  thought  it  important 
for  the  user  to  be  able  to  review  and  perhaps  change  responses  in  prior 
interactions.  Another  comment  cited  as  a  bad  feature  "being  stuck  in  a  loop," 
a  situation  in  which  the  system  rejects  a  response  but  neither  tells  the  user 
what  is  wrong  with  it  nor  allows  him  to  escape  to  on-line  help  or 
documentation  without  destroying  the  context  of  his  work. 

The  FCTC  OEs  were  receptive  but  not  enthusiastic  about  the  use  of 
graphics  in  interfaces.  They  saw  no  particular  advantage  to  graphics, 
especially  the  use  of  icons,  if  the  images  were  not  already  familiar  to  the 
user.  One  DE  thought  such  use  of  graphics  is  eye-catching  but  probably  no 
more  effective  than  alternative  approaches  requiring  less  expensive  hardware 
and  software.  In  addition,  use  of  graphics  does  not  preclude  errors  since  the 
user  must  still  remember  sequences  of  actions.  The  other  DE  was  also 
concerned  about  cost,  suggesting  that  graphics  are  probably  not  cost-ef fecti ve 
in  infrequently  used  systems.  Both  DEs  became  more  receptive  to  possible  uses 
of  graphics  when  the  interviewer  presented  a  hypothetical  task  of  specifying  a 
decision  network  for  sequencing  training  exercises  and  illustrated  (using  a 
pent i 1 -and-paper  protocol)  how  graphics  I/O  techniques  might  be  used.  They 
thought  the  graphic  display  of  the  network  as  a  graph  was  conceptually  useful. 
However,  they  did  not  feel  that  graphic  input  (via  a  mouse  or  other  pointing 
device)  would  necessarily  be  better  for  the  user  than  the  use  of  menus, 
especially  if  the  menus  and  graph  display  could  be  simultaneously  displayed. 
The  medium  and  organization  of  interaction  was  important  to  them  only  to  the 
extent  that  it  could  improve  the  amount  of  context  available  to  the  user. 

The  MPP.DC  DEs  had  a  different  perspective  on  user  interface  issues. 

Thei'-  comments  tended  to  reflect  the  current  common  wisdom  about  interfaces. 
They  also  contradicted  the  system  builders,  who  had  deemphasized  the  need  fur 
"user- friendly"  interfaces  for  experienced  and  motivated  DEs. 

The  NPRDf  DEs  believed  that  while  they  had  the  skills  to  use  arbitrarily 
complex  interfaces,  a  good  system  design  would  not  require  extensive  learning. 
The  ')[  who  had  worked  on  developing  a  database  for  CMS  stated  that  he  should 
not  have  had  to  learn  the  operating  system  interface  and  PASCAL  language  to 
attempt  that  task.  He  believed  that  characteristics  of  the  physical  interface 
increased  the  difficulty  of  the  conceptual  part  of  his  task.  Beyond  the 
physical  interface’s  shortcomings,  he  suggested  that  the  interface  should  have 
provided  conceptual  support  for  understanding  the  data  base  structure  and  its 
relationship  to  the  instructional  games.  He  felt  that  an  application  system 
for  DE-instructor  users  should  have  the  same  interface  features  usually  found 
in  training  systems  themselves:  turn-key  access  to  the  application,  built  in 
training,  initial  system  guidance  and  prompts,  function  keys,  and  descriptions 
and  manipulations  oriented  toward  the  organization  and  constructs  with  which 
the  user  conceives  his  task.  Several  other  comments  reflected  his  strong 
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belief  in  tailoring  system  capabilities  for  ease  of  use.  He  suggested  that 
external  documentation  must  avoid  "burying"  critical  details  aoout  critical 
basic  actions,  that  system  designs  should  trade  capabilities  off  to  improve 
learnability  and  effectiveness,  and  that  the  need  for  multi-media  and  multi- 
mode  user  interfaces  increases  as  the  capabilities  of  the  system  become  more 
varied. 

This  DE  also  provided  the  only  comments  about  g-  vhics  from  the  MPP.tJC 
respondents.  He  believed  graphics  are  only  impos  cant  when  they  reflect  solid 
analogies  wi  tin  the  user's  existing  understanding  of  the  task.  With  respect  to 
the  interviewer ‘ s  presentation  of  the  hypothetical  decision  network 
elaboration  task,  he  commented  that  direct  graphics  manipulation  via  a 
pointing  device  would  be  t he  best  input  mode  for  the  task  because  of  the 
ability  to  manipulate  structures  directly  and  ability  to  immediately  view 
results  of  the  manipulation. 

The  other  0!.  nad  similar  v  ws.  He  was  familiar  with  state-of-the-art 
"work  stations"  with  significant  built-in  user  interface  facilities.  He 
believed  that  hardware  has  been  the  main  limiting  factor  on  interface  quality 
and  tha^  work  station  technology  was  overcoming  these  limits.  He  cited,  for 
example,  the  original  NAVTAG's  inability  to  simultaneously  disp  a 
geographical  plot  arid  tabular  status  display  as  a  hardware  limit- Jon  on  the 
interface  that  adversely  affected  the-  system's  usability.  He  stated  that  ease 
of  initial  learning,  either  off-line  or  by  experimenting  with  the  system  was 
critical--be  thought  one  hour  of  introduction  price*  to  serious  use  was  a  safe 
maximum.  He  felt  two  design  features  would  insure  the  utility  of  these 
learning  sessions:  (1)  the  use  of  system  initial ’ve  ind  closed  input  modes 
(e.g.,  prompting,  menus,  function  keys,  and  pointing  devices) ,  and  (?.) 
transparency  (matching  the  structure  and  granularity  of  interface  interactions 
to  the  user's  conception  of  the  task). 

On  DE  knowledge  and  IK  AS  feasibility.  We  obt.j  n?il  only  limited  data  from  the 
Fou  For s  ~tnaT  oe  a  r  on  IK  A?  FeasibTTTty.  tHp  -ole  at  a  Of  in  building  a 
knowledge-based  system  was  unfamiliar  to  the  l  <'  T.  of  s,  so  they  were  able  to 
contribute  1,  .le  to  this  analysis.  The  IJF’RDC  >.s  were  familiar  with  that 
role  and  had  somewhat  more  to  contribute.  Trie  ,)t.s  emphasized  the  lack  of 
consensus  about  competent  performance  in  t.heir  speci  1 1  ties  (tactics  and 
acoustic  analysis).  One  rt'RPDC  HE  described  the  variability  in  experts' 
tactical  decisionr  w  ing  approaches  and  tolerance  for  that  variability  as  the 
result,  at  least  in  part,  of  the  tasks  characteristics:  uncertain  data,  a 
large  search  space  for  problem  solving,  and  the  heuristic  rialure  of  available 
procedures.  Since  no  one  is  ever  always  right,  alternative  approaches  are 
acceptable.  One  of  the  FLTC  UEs  attributed  expert  performance  variability  to 
the  competitive  nature  of  the  tactics  task  which  introduces  non-determinism 
and  prevents  a  stra ightforward  assessment  of  action  consequences. 

The  second  Nt’RHi.  coroner! tort  on  the  tacit  nature  of  expert  nnowlertge  in 
hi,  specialty.  Hi-,  experience  is  tha!  exports'  rationales  for  their  behavior 
do  not  coincide  with  the  behavior  tiny  exhibit.  To  induce  exports  to  give 
complete,  consistent  rationales,  it  is  necessary  to  confront  them  with 
contradiction-;  between  what  they  do  and  wnat  they  say  they  ao. 


The  DEs  confirmed  the  opinion  of  the  system  builders  that  they  do  not 
have  a  good  understanding  of  why  trainees  make  specific  errors  of  performance. 
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This  is  in  part  due  to  their  poor  understanding  and  consensus  about 
competence.  The  DEs  at  both  FCTC  and  NPRDC  asserted  that  in  actual  training, 
verbal  interaction  is  ordinarily  required  for  an  instructor  to  form  a 
hypothesis  about  a  trainee's  underlying  knowledge  deficiency.  One  NPRDC  DE 
believes  that  most  fleet-based  DEs  do  not  have  the  verbal  skills  for  such 
interactions  and,  further,  that  rank  differences  inhibit  fruitful  interaction 
during  training.  He  did  suggest  that  performance  diagnosis  from  behavioral 
data  might  be  easier  in  other  specialties  where  there  was  less  dependence  on 
uncertain  situational  data  and  on  data  interpretations  in  determining  which 
procedures  to  execute. 

The  other  NPRDC  OE  commented  that  most  DEs  have  little  opportunity  to 
develop  detailed  knowledge  about  trainee  errors  because  training  includes 
little  "over  the  shoulder"  monitoring.  One  of  the  FCTC  DEs  did  believe  he  had 
developed  diagnostic  knowledge  for  trainee  errors  from  his  experiences  at  that 
facility.  However,  he  acknowledged  that  the  nature  of  the  knowledge  was  such 
that  performance  data  from  many  exercises  would  be  necessary  to  converge  on 
diagnostic  hypotheses  about  a  trainee's  performance. 

Only  the  DEs  at  NPRDC  offered  comrtents  on  our  concept  for  an  IK  AS .  One 
noted  that  our  concept  of  a  class  generic  IKAS  was  consistent  with  his 
experience  in  sonar  interpretation,  where  there  are  about  five  different 
specialties  that  all  do  essentially  the  same  problem-solving  task  using 
di fferent  equipment  and  having  different  coordination  responsibilities.  The 
other  thought  that  automated  knowledge  acquisition  was  feasible,  but  that  any 
effort  to  develop  training  systems  would  probably  be  served  best  by  a 
combination  of  manual  and  automated  methods.  He  also  thought  that  an 
incremental  approach  to  developing  the  technology  would  be  critical  for  its 
success. 

One  mitigating  concern  was  that  if  development  of  knowledge-based  systems 
became  more  common,  there  might  be  a  dearth  of  DEs  with  appropriate  background 
and  motivation  to  assist  development  efforts.  In  particular,  unmotivated  or 
inexperienced  DEs  might  fail  to  contribute  to  the  development  of  a  KB  I S 
regardless  of  whether  knowledge  acquisition  was  manual  or  automatic  or  of  what 
user  interface  the  automated  system  had.  The  other  DE  shared  this  concern, 
stating  that  common  management  practice  of  assigning  personnel  rather  than 
soliciting  volunteers  could  negate  the  effectiveness  of  any  large-scale 
programs  of  knowledge-based  systems  development.  He  believed  that  suitable 
DFs  would  be  indifferent  to  whether  they  worked  with  a  human  KE  or  an 
automated  system  on  a  system-building  project.  That  is,  that  they  would  find 
working  with  an  intelligent  automated  system  acceptable. 

DISCUSSION  AND  RECOMMENDATIONS 

DISCUSSION  OF  INTERVIEW  RESULTS.  The  limited  and  informal  survey  of  system 
builders  and  DEs  can  only  be  generalized  with  care.  All  respondents  were 
members  of  two  small  Navy  communities  (other  than  the  Stanford  system  builders 
with  whom  we  interacted  informally  and  the  Navy  scientist  we  interviewed  by 
telephone).  These  communities  are  geographically  proximal  and  frequently 
interact  with  one  another.  The  DEs  represented  only  two  job  specialties  that 
are  somewhat  related  in  function.  On  the  other  hand,  they  represent  a  range 
of  roles  for  individuals  with  those  specialties  and  they  are  the  types  of 
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Individuals  who  would  most  likely  participate  in  R£[)  on  knowledge  acquisition, 
further,  the  DEs  and  system  builders  had  interacted  on  some  projects  and  had 
shared  other  experiences  involving  the  development  and  use  of  particular 
computer  systems.  Thus,  the  comments  we  received  reflect  perspectives  of 
System  builders  and  DEs  who  might  be  using  an  IKAS  together  today  if  one 
exi sted. 

The  responses  that  strike  us  as  most  important  are  those  indicating  that 
the  effectiveness  of  automated  knowledge  acquisition  depends  first  on 
selecting  appropriate  DE  users,  regardless  of  the  system’s  user  interface. 
Appropri ateness  seems  to  refer  to  the  DEs  motivation  and  basic  understanding 
of  computers.  Since  we  were  told  that  most  DEs  of  officer  rank  have  the 
latter,  it  seems  that  motivation  is  the  key  factor.  By  itself,  a  user 
interface — no  matter  how  user- friendly — will  not  guarantee  that  an  arbitrary 
DE  will  become  a  productive  member  of  a  knowledge-based  system  development 
effort . 

Beyond  tins  joint,  we  observed  considerable  divergence  of  opinion.  The 
system  builders  believe  that  no  particular  physical  user  interface 
character! sties  are  required  for  a  motivated  DE  collaborating  in  a  development 
effort.  Their  experience  indicates  that  such  DEs  are  willing  and  able  to 
learn  to  use  the  sane  physical  user  interfaces  they  themselves  use.  They  do 
believe  that  any  system  must  provide  conceptual  support  for  the  user.  Thus, 
the  "what"  of  human-computer  interaction  is  important,  but  the  "how"  is  not. 
The  DEs  expect  such  conceptual  support,  but  they  also  want  a  friendly  physical 
interface  to  the  system.  Although  they  can  and  do  learn  to  use  systems  with 
idiosyncratic  and  complicated  user  interfaces,  they  find  this  to  be  an 
imposition  and  an  impediment,  to  achieving  their  goals  in  using  those  systems. 

The  DEs  expressed  no  consensus  on  what  features  they  desire  in  a  user- 
friendly  interface.  Like  the  system  builders,  they  recognized  that  tradeoffs 
exist  and  that  interface  design  depends  on  objectives  and  functional 
capabilities  of  the  system.  The  Oils  were  largely  noncommittal  about  specific 
media  or  organizations  of  the  interaction.  They  were  more  concerned  with 
issues  of  conceptualization  rather  than  how  these  should  be  resolved  at  the 
level  of  input  and  output  implementation.  For  example,  the  DEs  indicated  that 
they  thought  graphics  I/O  methods  would  be  of  real  value  only  when  the  user 
already  had  a  spatial-visual  framework  for  conceiving  t.he  problem  domain. 

There  was  only  one  consistent  media- related  constraint  mentioned:  that  minimal 
typing  should  he  required  to  achieve  desired  functionality. 

The  higher-order  issues  the  DEs  commented  on  were: 


a.  Reseono  v'oess.  Poor  responsiveness  may  inhibit  effectiveness 
for  at  least,  seme  users.  It.  was  not  clear  from  t.he  comments  whether  th< 
main  problem  would  be  in  impeding  the  user’s  desired  rate  of  interaction 
or  in  confusing  or  irrifatiny  him  by  slow  or  variable  feedback  to  his 
inputs. 

b.  t 1  ex i ! i i 1 i ty .  A' fornati ve  or  rich  modes  at  input  are  unnecessary 
because  FTavy  personnel  are  ace  listened  to  schematic  fixed  artificial 
languages.  1  l--x  ioi  1  i  fy  in  handling  errors  is  desirable. 
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c.  Context.  Preservation  and  display  of  prior  interactions  is 
useful.  ^ainTenance  of  context  across  errors  and  user  exercise  of 
initiative  is  critical.  There  were  no  consents  regarding  the  degree  of 
context  preservation  desirable  between  work  sessions. 

d.  User  Control.  The  two  groups  of  DEs  had  somewhat  divergent 
views  about  user  control .  The  difference  may  reflect  whether  the 
respondents  were  focusing  on  introductory  or  repeated  use  of  a  system. 

The  NPRDC  DEs  favored  considerable  system  control  of  interactions.  They 
seemed  to  be  considering  initial  system  accessibility  and  conceptual 
support  for  a  user.  The  FCTC  DEs  favored  global  user  control  of 
interactions  with  system  control  exercised  locally  within  particular 
types  of  interactions.  They  seemed  to  be  considering  long-term  use  of 
systems  and  their  opinions  may  have  been  influenced  by  experience  with 
systems  having  poor  implementations  of  system  control.  Both  groups 
believed  that  user  control  to  escape  a  system-driven  interaction  is 
necessary.  Much  of  their  concern  with  context  management  centered  on 
supporting  this  type  of  user  control. 

e.  User  Knowledge  Requirements.  The  DEs  thought  that  a 
knowledg*ea"5Te  "user"  ought  to  be  abl e  to  use  a  new  system  effectively  in 
approximately  one  hour.  To  do  so,  they  believed  that  off-line 
documentation  must  be  organized  effectively  and  that  on-line  help  should 
always  be  available.  Their  strongest  comments  regarding  user  knowledge 
requirements  concerned  transparency.  They  believed  that  transparency  is 
instrumental  to  reducing  the  amount  of  new  knowledge  required  to  access  a 
system.  We  see  this  as  a  further  statement  about  the  need  for  conceptual 
support  if  a  system  is  to  be  used  effectively. 

Most  of  the  comments  collected  on  the  feasibility  of  an  IKAS  came  from 
the  system  builders.  They  all  believed  that  a  generic  IKAS  for  knowledge- 
based  systems  is  not  currently  possible  nor  may  it  ever  be.  Their  beliefs, 
like  ours,  derived  from  the  perceived  dependencies  among  system  objectives  and 
capabilities,  required  knowledge,  representation  formalisms,  and  knowledge 
elicitation  methods.  Some  system  builders  thought  that  IKAS  development  with 
more  limited  applicability  was,  however,  a  worthwhile  goal.  Such  research 
could  aid  future  system  building  and  maintenance  efforts.  More  fundamentally, 
it  could  contribute  new  knowledge  to  the  field  of  knowledge-based  systems 
technology. 

Tommenfs  from  most  system  builders  and  Df's  supported  the  concept  of  a 
•  •  1. 1 ss- generic  IKAS  .is  a  feasible  focus  for  research  on  techniques  to  support 
knowledge  acquisition.  They  agreed  that  the  concept  of  classes  of  tasks  was 
at  least  intuitively  viable.  The  system  builders  believed  that  there  are 
difficult  problems  to  be  solved  if  the  IKAS  system  is  to  use  class-generic 
knowledge  to  support  its  i nteracti ons,  but  that  solutions  to  these  problems 
are  possible  given  the  current  state-of-the-art. 

The  greatest  concern  about  the  IKAS  concept  involved  the  question  of 
whether  DEs  have  and  can  articulate  the  types  of  knowledge  an  IKAS  would  be 
designed  to  obtain.  There  was  consensus  among  the  system  builders  and  DEs 
‘hat,  at  least  in  the  specialties  with  which  they  are  familiar,  DEs  do  not 
have  good  causal ,  diagnostic  knowledge  associating  performance  errors  with 
knowledge  deficiencies.  The  system  builders  also  questioned  DE  abilities  for 
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formulating  instructional  interventions  icr.’!  «■  t.r  the  individual  re-- 
trainees.  The  rtfs  were  confident,  however ,  mat  .  nev  inn  :,k  i  >  ot-er 
knowledge  about  some  errors  that  might,  be  observed  our m-g  performance  m.v 
explanations  for  those  errors  in  terms  of  pc./-.',  <? ■:  matures. 

RE  COMMEND  AT  I OKS  FOR  IK  AS  DESIGN.  As  we  have  stater,  ;  be  at  flity  tc  spe 
tne  design  of  an  IKAS,  particularly  its  u-,er  interlace,  depends  on  the 
capabilities  and  architecture  of  the  host  knowledge- based  instructional 
to  be  served  by  the  IKAS.  The  recommendations  we  :  an  offor  1  ,  ‘  a  •■.on 

interactions  with  OEs  and  system  builders,  are  thus  1  ini  ted  .  m 

should  be  considered  when  the  other  requirements  and  constraints  for 
developing  a  particular  design  are  obtained. 

a.  Emphasize  conceptual  support  and  transparency .  The  IKAS  shouiu 
help  the  tit  -user  recognize  or  Hearn  aho'u  t”Tts"  relationship  to  the  host 
instructional  system,  its  mode"  of  the  use'..  task  in  ir.tora^ti  g  with 
it,  and  its  intended  model  for  describing  the  knowledge  about  the  user's 
area  of  expertise.  Its  interactions  should  conform  tc  the  Vve'i  of 
abstraction  defined  by  these  f ramewor.. s  and  mo  dr  1  s  and  hide  i c; wo r- level 
implementation  details  from  the  user.  A  partial urly  important  form  of 
conccotua '  support  is  to  enable  the  user  to  anticipate  how  knowledge  he 
specifies  to  the  IKAS  will  manifest  itself  in  the  host  system*  s  behavior . 

b.  Design  the  global  knowledge  acquisition  strategy  to  be  modular 
with  respect  to  'achtevTng  aVqu"" sTtTon  oTTff’rTe’renT  fyj>e~  of  knowledge”." 

r  ir  some  spec  fa  1  ties’  or  tor  an  T3F  within  .T~gTvVn  'soec  !*aTty',  11'fiTn  types 
of  knowledge  may  not  be  known  to  the  user  (e.g. ,  rules  far  ,iusal 
diagnosis  of  per fcmiance  errors) .  The  system  should  be  able  to  oh tain 
otner  types  of  knowledge  from  the  DE  and  leave  the  definition  of 
knowledge  the  DE  cannot  articulate  to  other  methods. 

c.  Provide  on-line  help/document,  its  or-  fan !  iv'es  arid  intelligent 
context,  'management.  These  "i  n  ter  face"  Tac  fi  i  tTes ’seeiii’  most  Trap'. iff  ant  for 
i'nsurT ng  accessi bi  1  i ty  and  effective  use  by  relatively  coniputer- 

siu  ni sti rated  DEs,  such  as  those  interviewed. 

u.  "'efer  commitment's  on  other  user  irtirE .  '"'.us  i;  .« 

tin  ►••‘mental  design  and  deveT comer t  ararcfi.l.  ’  *•  > ,  T  .cif-  .*  ’ha  Dr 

i  nd  1 1.  at  ”d  a  yreferrm  e  and  need  for  a  a  F  ■ :  ■  ’  •.  .  -  t  r  j  eiiril  y  interface, 
they  did  no*  d  f --••  an  Integrated  picture  .,[  w>  it  I  nut  would  'I  rice 

they  ii.iv--  i-i.in  i  fed  their  ability  tii  io.f  (Ci  :  1  a  1  nterfa.s  they 

.hnuld  ho  able  is  use  most  systems  as  long  as  they  deliver  some 
r one ep tua 1  supper t.  We  hoi iev»*  that  the  best  approach  to  developing  the 
i  n  tert'di  f>  d-'.ign  is  through  managed  refinement  of  a  flexible  m  o  to  type, 
based  oe  inferaciion  w - th  DPs  using  that  prototype.  This  approach  allows 
r.  empi "  it /.•  1 '  y  ba.ed  resolution  of  the  tradeoffs  between  features  and 
./  stem  re  soon-  ’vines.  The  approach  ultimately  re  go  i  **es  p  )totyp>0 
P'V" 1 opment  us  <  ng  hardware  and  software  with  potent i a ’  mul t- -media 
rapah’lit’es  to  support  alternative  input  modes  and  nodes  or  omanizat4 on 
for  interactions — for  example,  h  i  gh-  performance ,  ingle-user  vr.irk 
•stations  with  high-resolution  graphics  and  multiple  input  media. 

However,  initial  work  on  conceptual  support  and  on  higher-level  interface 
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functional ity  could  proceed  with  less  extensive  hardware-software 
resources  using  a  more  modest  and  limited  physical  interface. 
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SECTION  VI 

THE  IK  AS  ARCiilTtCTURL 


This  sect  inn  presents  a  high-level  architecture  Tor  imp!  eventing  the  IK  AS 
concept  introduced  in  Section  IV.  Toe  archi recture  was  designed  to  support 
the  most  useful  features  developed  in  prior  effort-  t.r.  support  know’ •'■dgo 
acquisition  (see  Section  III)  and  to  satisfy  the  r-qui '-caents  derived  from  our 
discussions  j i  in  Navy  personnel  (see  Section  7; . 

The  structure  and  level  of  detail  of  me  arch  i  tec  tune  is  compatible  with 
all  three  alternatives  for  IK AS  development  described  in  Section  IV.  In  fact, 
it  could  serve  as  a  framework  -or  any  stable,  class-generic  knowledge 
acquisition  system,  instructional  or  otherwise.  Any  cure  detailed 
specification  of  the  areni lecture  would  require  nv initmcnts  to  snecific 
representation  forma!  ism,  use  -  interface  mechanisms,  -vd  other  n.echari  sms 
dependent  on  th<>  noma  in  character i sties  end  capabilities  ot  the  host 
knowledge-based  system  it  would  serve, 

► igure  -  shows  the  major  functional  modules  the  design  and  general 
descriptions  n!  the  information  they  exchange .  To  f;-st  discuss  each  module's 
function  the  >>'< a  i  t.  turn  and  t.en  dim  ss  tic  design  constraints  the 
arc  hi  lecture  a  mi.,  to  satisfy .  For  each  ot  the  three  user-accessible  modules 
( the  El  ir.;  TOR,  tiv'  '  vi TOR,  and  t in o  EXERCISER',,  we  provide  examples  of  the 
functions  f.,r  tin  threr,  jkaS  concepts  described  in  Section  IV. 

IK  AT  MO.FJLt.S 

ft  IT  I  ','P.  ’he  EilCiros  generates  suggestions  and  requests  for  types  of 
knowledge'  to  be  <?!  i-  <  ted  from  the  user  when  the  system  is  controlling  tr,e 
interaction.  T t  dor s  ■'  hv  opera t i nal  i g -. -v.  .,-i  agenda  '..rpateii  by  the  f’l  AViEE, 
converting  t r. •  agenda  specifications  uito  req.n'srs  to  be  sent  fr>  the  user 
across  the  user  interface,  and  then  delivering  these  requests  either  by  its 
own  median ;  sms  or  those  of  the  ASSISTANT.  Agenda  specifications  woulo 

include  :;»v.*rai  type:  u.  t  i  v>  ties.  The  FLIC  IMP  mignt  suggest  a  focus  si 

attention  on  s>mv  >■,  ( •:  • ,  dime us  :on  of  the  know  tenge  h,;,e.  AI  ternat 4  vel  v, 
it  ([light  r  r  <  j ; ; ,  ■ ;  i  m.  ?  i ,  1  ir  input-  required  for  (a!  s  t.r  in  tur.j1  couple  tenons  :,.- 

( h!  < ■  i ••  i r  i  • . i , . , , ,  y  with  ’  t,<-  '.emanti,  and  pragmatic  features  of  ‘die  dosn i n  and  ot 
thi'  luior  lint''.-,  it,.  ■  i:  !  i'M'  might  also  suggest  appropr  i  a  t  r  times  f;.>-  the 
user  to  i  nvnke  the  ;  >;!  to,  t  si  I’  to  test  the  effects  :>f  modi  fi‘ iti  ons  to  the 
k  nnwl  ed  j,  ■  h,t  \r  . 

'-,t  F.  I  ‘ .  T  1 1  s'sji-,  he  able  to  explain  its  hehav  I  ir  to  the  user  using  a 
ntionale  created  along  with  trie  agenda.  The  user  con  ask  why  the  FI.  Id  TOR  i; 
following  the  r.  irrent  tack,  and,  on  the  basis  of  tne  response,  dec i de  whether 
h>  warts  to  .  otiv,  i  y  0r  take  the  initiative  himself.  '  ,>r  instance,  the  d  li.il. ‘t 
may  be  able  tn  •,  the  user  it  is  asking  fee-  types  of  knowledge  in  a 
particular  order  because  that,  order  proved  «ffecti ve  in  eliciting  knowledge  in 
Other  domains. 
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MANAGER  ->  EXERCISER 

Pointers  to  information  in  the  knowledge  base 
RECORDER  ->  MANAGER 

Information  pertinent  to  knowledge  base  checkpointing 
a”;!  at -..I  1  log  decisions 

MANAGER  CHECKER 

Pointers  *  ■' •: forma t i on  in  the  knowledge  base 

MANAGER  ->  R|  ANNER 

Pointers  to  information  in  the  knowledge  base 
MANAGE  R  ->  r  1. 1 C  I  TOR 

Pointer-,  ti  information  in  the  knowledge  base 

MANAG:  k  ■  :  .  ■ :  TO- 

Pn  j  -iters  ■-  information  in  the  knowledge  base 

fi  !C!’wP  —  RETfPOER 

"se.f-5  r,  fe-n  i t. tr -3C  t  i  on  ryCoM 


RECORDER  ANNLP 

Records  t,{  rect-rt  interaction  contest 

PLANNr  P  -  •  UK  II  OR 
Elicitation  agenda  and  rationale 

Checker  ->  e.mcitor 

Results  of  knowledge  chock 

CHECKER  ->  EDITOR 
Results  of  knowledge  check 

RE f  '  Pile  R  ->  Cat  K.  lSEt 

Prior  exercise  definitions  and  results 

n.!f  :t'-p  checker 

Knowl>-  ige-base  modification  to  be  checked 
EDITOR  ->  fWECKfR 

Knowledge- base  modification  to  be  checked 

I  DITOR  ->  Rf :  IP::' -i 
i n to rac t i or  records 

rxfRCIGFR  ->  RECORDER 
Exercise  records 

El  IC  I  TOR  ->  MANAGER 

Request  for  knowledge  base  pointer 

changes  to  current  knowledge  base 

EDITOR  ->  MANAGER 

Request  for  knowledge  base  pointer 
Changes  to  current  know'edge  base 


figure  ’<  (rout..).  iKAE  ,ir-  f:  1 1  nr  1  urn 
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s  EDITOR  ->  EL IC ITOR 

User  response  to  EL  IC ITOR  query  achieved  via  EDITOR  use 

t  EL IC ITOR  ->  USER  ASSISTANT 

Queries  to  user 
Error  information 

u  EDITOR  ->  USER  ASSISTANT 

State  information 
Frror  information 

v  EXERCISER  ->  USER  ASSISTANT 

State  information 
Exercise  results  for  user 
Error  information 

w  USER  ASSISTANT  ->  EL IC I  TOR 

User  responses  to  information  requests 
User  requests  to  restore  prior  state  or 
modify  prior  Inputs 

X  USER  ASSISTANT  ->  EDITOR 

User  commands 

User  requests  to  restore  prior  state  nr 
modify  prior  inputs 

y  USER  ASSISTANT  ->  EXERCISER 

User  requests  t.o  test  system  performance 

z  HELP  FACILITY  ->  USER  ASSISTANT 

Pointers  to  on-line  documentation 

aa  HELP  FACILITY  ->  CAI  FACILITY 

Pointers  to  on-line  documentation 

bb  HELP  FACILITY  ->  USER  WORK  STATION 

Documentation  requested  by  user 

CC  USER  ASSISTANT  ->  USER  WORK  STATION 

Information  passed  by  ELICITOR,  EDITOR,  EXFRCISER 
Responses  to  requests  and  commands  trapped  by 
USER  ASSISTANT 

■id  CAI  FACILITY  ->  USER  WORK  STATION 

Instructional  content 

PC  USER  WORK  STATION  ->  HELP  FACILITY 

Requests  to  examine  system  documentation 

ft  USER  WORK  STATION  ->  USER  ASSSITANT 

Inputs  to  be  passed  to  ELICITOR,  EDITOR,  EXERCISER 
Commands  and  responses  to  USER  ASSISTANT 

<)i|  USER  WORK  STATION  ->  CAI  FACILITY 

Inputs  to  instructional  interactions 


figure  2  (cont.).  IKAS  architecture 
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a.  Alternative  2  •  The  F  L.  I C I  TOR  can  determine  c'  suggest  whether 
the  user  sHotJ! <f Toctis  an  (1)  sequentially  describing  performance 
characteristics,  situation  characteristics ,  or  mappings  of  observables  to 
deviations  from  the  competence  model,  or  (?)  all  of  these  for  successive 
segments  of  the  competence  model.  It  can  ask  about  situation 
characteristics  when  particular  performance  character! sties  have  been 
specified.  It  can  suggest  tnat  the  user  invoke  the  EXERCISER  when  an 
unfamiliar  form  of  variation  from  the  competence  monel  has  been  specified 
as  an  explanation  of  a  set  of  performance  characteri sties. 

b.  Alternative  2.  Tne  EL  T  C I  TOR  can  determine  or  suggest  whether 

el ici tation"  sho'uTa'be  organized  according  to  situations  or  to  procedures 
and  ^ules  in  the  existing  opponent  simulation.  U'hen  focusing  on  a 
particular  situation,  it  can  ask  about  whether  specific  rules  and 
procedures,  applied  in  "similar"  situations,  might  also  be  applied  in 
that  situation. 


Alternative  3.  Inc  Q.fCITOR  can  determine  or  suggest  whether 
' '  i.;  ;  fatnin  should  pursue  elaboration  of  taxonomies  or  the  attributes  of 
the  concepts  •>'  the  taxonomies.  It  can  ask  whether  features  associated 
with  a  rorirept  are  also  associated  with  concepts  that  are  "close"  in  the 
taxonomic  structure.  It  can  suggest  that  the  user  invoke  the  EXERCISER 
when  a  concept  with  new  types  of  attributes  is  described. 

EDI  TOP..  The  EDITOR  allows  the  user  to  builc,  modify,  and  inspect  the 
knowledge  base  under  his  own  initiative,  it  is  also  available  through  the 
EL  I r I  TOP  as  a  mode  for  responding  to  seme  requests  for  selection  of  elements 
in  the  knowledge  base  and  for  description  of  modifications  to  knowledge  base 
structure . 


The  EDITOR  supports  transparent,  structure-oriented  speci fi cation  of 
attention  and  modi ficati on .  That  is,  the  granularity  of  its  commands  is 
consistent  with  the  syntax  and  semantics  of  the  knowledge-base  formalisms,  «v.t 
the  low-level  software-  and  hardware-dependent  implementations  of  those 

•  onaul  ism--..  1  he  KAS  (Reboh,  19R> )  network  editor  is  a  good  mode!  for  such 

functionality.  Additional  support  for  handling  syntactically  invalid  commands 

•  unpl  fed  v'ci  open  input  mode  is  obtained  through  the  IP  I  TOR '  s  interface  with 
the  I'Si.R  ASS ! Si A‘.ir ,  while  support  for  handling  semantic  and  pragmatic  errors 
>'s  obtained  from  the  CHFCKER.  error  checking  is  performed  on  each  input  to 
provide  the  user  immediate  feedback. 

The  fbirbR  ilso  must  provide  context,  display  of  the  local  focus  of 
attention  within  the  global  knowledge  base.  If  the  representation  formalisms 
for  a  domain  class  are  conducive  to  graphic  display  and  manipulation,  context 
display  might  comprise  dynamic  "maps”  cf  the  knowledge  base  displayed 
concurrently  with  the  local  editing  window.  In  addition  to  context  display, 
t.he  EDI  TOR  must  also  enable  the  user  to  locate  existing  knowledge  and  change 
attention  Oy  specifying  oartial  descriptions.  Other  context  management 
functions,  such  as  maintenance  and  display  of  alternative  contexts,  are 
handled  by  mechanisms  external  to  the  EDITOR. 


Examples . 
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a.  Alternative  1.  A  schema-based  editor  is  possible  for 
descriptions'  oT  performance  and  situation  characteri sties  embedded  within 
a  network  editor  if  those  descriptions  are  hierarchical.  The  frames  and 
schemas  are  manipulate  as  slots  and  values  of  defined  types.  The  syntax 
of  each  defined  type  determines  the  granularity  of  access  to  values  of 
that  type.  A  network,  procedure,  or  rule-based  editor  would  be  used  for 
elaborating  deviations  of  the  competence  model,  depending  on  how  it  is 
represented. 

b.  Alternative  2.  An  editor  allowing  manipulation  of  networks, 
procedures,  and  ruTes  would  be  used  for  elaborating  al ternaM ves  to  the 
opponent  simulation  model.  A  network  editor  would  allow  manipulations  on 
nodes  (procedure  or  rule  designators)  and  links  (control  paths)  to 
specify  deletion,  insertion,  and  reordering  of  node  invocation.  A 
procedure  editor  could  be  oriented  toward  modification  of  defined 
procedures  rather  than  composition  from  scratch.  A  rule-editor  would 
allow  the  user  to  manipulate  the  specification  and  logical  composition  of 
primitive  clauses  in  the  condition  and  action  portions  of  the  rules 
governing  opponent  behavior.  The  syntax  of  the  clauses  would  be  used  to 
determine  the  granularity  of  access  to  their  components  (e.g.,  as  members 
of  tuples  in  predicate-object-value  conditions). 

c.  Alternative  3.  A  network  editor  would  be  used  to  specify 
concept  aruTTeYture"  taxonomies.  It  could  be  combined  with  a  schema-based 
editor  to  permit  specification  of  concept  attributes  and  attribute 
values.  A  graphics  interface  could  be  used  to  support  the  network  editor 
and  would  be  advantageous  for  its  context  display  capabi 1 i ties. 

EXERCISER.  The  EXERCISER  gives  the  user  access  to  the  host  KBIS.  Its  majo1- 
function  is  to  provide  conceptual  support  through  feedback  about  the 
relationship  between  user  modifications  of  the  knowledge  base  and  behavior  of 
the  KBIS. 

A  variety  of  EXERCISER  capabilities  might  be  implemented.  Most  simply, 
the  IKAS  user  may  be  allowed  to  access  the  KBIS  through  its  instructor  or 
student  interfaces  to  assess  system  behavior  as  he  changes  the  knowledge  base. 
Additional  control  over  configuring  the  state  of  the  KBIS  would  allow  the  user 
to  directly  configure  and  test  an  "interesting"  situation  he  wants  to  examine. 
As  in  some  expert  consultation  systems,  the  user  could  draw  from  a  library  of 
problem  cases  or  use  an  editor  to  alter  these  cases  in  order  to  exercise  the 
KBIS  and  obtain  summaries  of  results.  In  addition,  the  EXERCISER  could  be 
used  to  automati cal ly  select  entries  from  the  library  according  to  a  set  of 
heuristics.  These  heuristics  would  seek  to  thoroughly  exercise  new  or 
modified  knowledge  entered  by  the  user.  The  requirements  for  and  feasibility 
and  implementation  of  these  EXERCISER  capabilities  are  a  function  of  the  host 
system's  capabilities  and  implementation. 

F  xampl os . 


a.  Alternative  1.  The  EXERCISER  could  be  used  to  determine  the 
range  of~behaviors  classified  with  a  given  set  of  performance 
characteristics.  After  postulating  certain  errors  and  circumstances  in 
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which  they  should  occur,  the  user  could  observe  performance  of  the  model 
on  problems  for  which  the  anticipated  errors  should  occur.  He  could  also 
test  the  reliability  of  performance  diagnoses  as  the  knowledge  base 
grows. 

b.  Alternative  2.  The  user  could  observe  performance  of  variants 
of  the  opponent  sTmuTatioo  in  different  ta>  tical  situations.  He  could 
also  observe  the  performance  of  all  the  variants  in  a  particular 
situation.  Thus, the  EXERCISER  could  be  used  to  determine  the  range  of 
behaviors  ti.j  opponent  could  exhibit  and  the  conditions  under  wlv'ch 
different  behaviors  would  be  invoked  by  the  KBIS. 

c.  Alternative  3.  The  user  could  observe  use  of  his  defined 
concepts  in  tfffferent  instructional  games.  He  could  observe  the  games1 
sequential  behavior  for  a  set  of  concepts.  In  each  case,  he  could 
examine  the  selection  rules  used  by  the  games  in  accessing  the  taxonomies 
and  concept  definitions. 

RECORDER.  The  RECORDER  acts  as  a  common  store  of  recent  context  about 
activities  in  the  ELlCiTfJR,  EDITOR,  and  f XLR.'.IGER.  fni s  information  supports 
the  PLANNER  in  its  maintenance  of  an  agenda  for  inter  -lotion  topics.  It 
provides  that  same  context  to  the  EXERCISER  to  support  any  bookkeeping  or 
other-  EXERCISER  functions  that  depend  on  knowledge  of  interaction  history. 

The  nature  of  the  stored  "context"  is  dependent  on  the  functions  of  the 
ELICITOR,  EDITOR,  and  EXERCISER.  These  will  vary  from  application  to 
application.  Generally,  "context"  refers  to  the  focus  of  attention  within  the 
knowledge  base,  what  effects  were  achieved  at  that  focus,  and  how  they  were 
achieved. 

The  RECORDER  also  provides  results  of  EXERCISER  invocations  to  the 
MANAGER.  These  results  are  used  by  the  MANAGER  to  make  decisions  about 
archiving  of  the  knowledge  base  in  physical  storage. 

PLANNER.  The  PLANNER  generates  the  agenda  used  by  the  ELICITOR.  It  user, 
heuristic  rules  that  operate  on  the  knowledge  base  and  the  information 
maintained  by  the  RECORDER  to  produce  a  set  of  proposed  topics  and  user 
queries  to  stimulate  the  growth  oF  the  knowledge  base.  The-  PLANNER'S  rules 
embody  knowledge  (acquired  by  the  KEs  who  engineer  the  first  Ki.iS  for  the 
domain  class)  about  the  syntactic  structure  and  cl  ass-generic  semantic  and 
pragma  Hr  knowledge  of  the  (Pimuin.  This  knowledge  can  be  used  to  define 
dialogue  t/ieues  and  transitions  that  are  sensible  in  terms  of  their  conceptual 
relationships  and  importance .  They  also  embody  knowledge  about  more  general 
aspects  of  dialogue  management— for  example,  the  need  to  vary  content  and 
style  to  avoid  user  boredom. 

Since  the  IK  AS  design  is  oriented  toward  a  mixed-initiative  approach  to 
control,  the  PLANNER  needs  to  plan  an  agenda  only  for  a  short  time  horizon. 

As  items  from  the  agenda  are  executed,  the  PLANNER  is  invoked  whenever  the 
user  has  problems  fulfilling  the  ELTCITOR’s  requests  or  assumes  the  initiative 
by  invoking  the  EDITOR.  At  that  point,  the  PLANNER  replans  the  agenda.  If  the 
user  ihoul  1  complete  the  local  agenda,  then  the  PLANNER  is  invoked  to  plan  for 
another  short  time  horizon.  This  dynamic  planning  and  replanning  for  a  snort 
time  horizon  insures  that  the  FLICITOR's  behavior  is  always  sensitive  to 
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recent  events.  In  particular  the  ELICITOR  should  follow  up  on  episodes  of 
user  initiative. 

The  PLANNER  saves  a  rationale  for  the  agenda.  The  rationale  describes 
for  each  agenda  element  which  rule  or  rules  caused  it  to  be  included  and  what 
features  of  the  knowledge  base  or  context  caused  those  rules  to  be  applicable. 
This  is  similar  to  the  rationale  provided  by  rule-'ased  expert  consultation 
systems.  Tne  rationale  can  be  used  by  the  ELICITOR  to  justify  its  behavior  in 
response  to  user  request.  The  ability  to  supply  rationales  should  make  the 
user  more  confident  in  the  ELICITOR' s  behavior  and  provide  information  the 
user  can  weigh  in  deciding  whether  or  not  to  invoke  the  EDITOR  to  work  under 
his  own  initiative. 

CHECKER.  The  CHECKER  provides  error  checking  for  knowledge  base  modifications 
entered  through  the  ELICITOR  and  the  EDITOR.  It  detects  and  reports  to  those 
modules  structural  and  semantic  anomalies  in  the  specifications.  Such 
anomalies  may  include  a  modification  that  would  make  a  taxonomy  circular,  a 
value  for  an  attribute  that  is  logically  inconsistent  with  the  value  for  a 
related  attribute,  or  a  rule  that  will  never  execute  because  its  activation 
conditions  are  subsumed  by  other  rules.  The  use  of  closed  input  modes  by  the 
ELICITOR  may  preclude  some  such  errors  from  occurring,  and  the  ELICITOR  and 
EDITOR  may  perform  some  error  checking  on  their  own.  However,  such  checking 
will  be  local  to  the  focus  of  attention.  Generally  speaking,  the  CHECKER'S 
role  is  to  provide  more  global  checking  against  the  existing  contents  of  the 
knowledge  base  to  prevent  modifications  that  would  produce  errors  when 
considering  the  overall  definition  of  the  knowledge  base.  This  capability 
requires  that  the  CHECKER  incorporate  meta-knowledge  about  the  structures  and 
syntax  of  the  knowledge  base.  Such  error  checking  has  been  a  feature  of  the 
knowledge  acquisition  support  of  several  expert  consultation  systems  (see 
Section  III). 

In  order  to  avoid  the  compounding  of  errorful  specifications,  the  CHECKER 
operates  on  the  contents  of  each  modification  entered  via  the  ELICITOR  and 
EDITOR.  Following  such  checks,  the  EDITOR  and  ELICITOR  forward  modifications 
to  the  MANAGER.  Information  about  CHECKER  rejections  is  forwarded  to  the 
RECORDER  sine0  it  may  be  of  use  in  planning  the  elicitation  agenda. 

MANAGER.  The  MANAGER  frees  the  user  and  the  other  modules  from  the 
responsibility  of  coordinating  the  management  of  the  knowledge  base  and  its 
physical  storage.  Rather  than  operate  on  separate,  local  copies  of  the 
knowledge  base,  the  modules  share  a  single  virtual  knowledge  base  maintained 
by  the  MANAGER.  This  centralized  management  appears  efficient  for  supporting 
mixed-initiative  elicitation  where  both  the  ELICITOR  and  the  EDITOR  are  us°d 
in  an  interleaved  manner. 

The  MANAGER  must,  maintain  a  chronology  of  knowledge  bases  so  that  the 
El  I Cl T OR  or  the  user  (through  the  EDITOR  or  EXERCISER)  can  access  and  perhaps 
restore  prior  contexts.  The  objective  is  to  allow  the  user  to  examine  and  use 
prior  or  alternative  knowledge  bases  and  to  protect  the  user  from  loss  due  to 
system  errors.  The  physical  storage  state  of  earlier  knowledge  bases  and  the 
method  for  representing  successive  variations  is  invisible  to  the  modules  and 
user  and  depends  on  knowledge  the  MANAGER  has  about  the  underlying  operating 
system  and  hardware.  The  cost  of  the  capability  to  restore  any  arbitrary 
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prior  state  would  appear  too  high.  In  the  short  term,  the  RECORDER  and  USER 
ASSISTANT  may  contain  information  sufficient  to  restore  to  any  rece  t  context, 
but  in  the  longer  term  some  intermediate  state  information  needs  to  be 
discarded.  The  MANAGER  preserve,  snapshots  of  the  knowledge  base  at  points 
specified  by  the  user  (through  the  EDITOR),  the  ELICITOR,  and  the  RECORDER. 
Those  modules  include  heuristic  rules  for  determining  critical  junctures  at 
which  a  complete  long-term  record  of  context  may  be  required.  The  MANAGER  may 
also  determine  requirements  for  saving  a  full  context  based  on  its  information 
about  system  state. 

USER  ASSISTANT.  The  USER  ASSISTANT  monitors  and  supports  all  of  the  user's 
interactions  with  t.ne  ELICITOR,  EDITOR,  and  EXERCISER.  It  is  modeled  roughly 
on  the  capabilities  for  lov  level  user  support  provided  in  the  INTERLISP 
programing  system  (Tei telman ,  et  al ,  1978).  These  capabilities  include 
correction  of  spelling  and  other  simple  syntax  errors;  examining,  redoing,  and 
undoing  recent  events;  support  for  user-defined  procedures  and  abbreviations 
("macros");  and  ready  access  to  context-sensitive  help  documentation.  These 
capabilities  reduce  the  consequences  of  errors  and  allow  the  user  to  operate 
more  efficiently  by  focusing  or.  hi?  conceptual  task  rather  than  low-level 
communication  with  the  system.  Instead  of  supporting  each  of  these 
capabilities  within  the  user-accessible  modules,  they  are  achieved  uniformly 
within  the  USER  ASSISTANT  in  order  to  promote  greater  consistency. 

The  USER  ASSISTANT  operates  by  trapping  all  I/O  with  the  user,  examining 
it  to  determine  whether  any  of  its  procedures  are  applicable,  recording  it, 
and  passing  it  on.  Its  capabilities  depend  on  its  knowledge  about  the 
subsystems  with  which  the  user  interacts  (e.g.,  spelling  correction  lists, 
syntax  for  open  input  modes,  inverse  operations  for  undoing  prior  events,  and 
pointers  to  information  in  the  help  facility). 

HELP  FACILITY.  The  HELP  FACILiTY  is  a  documentation  database  for  the  IK AS 
used  by  the  user,  the  CAI  FACILITY,  arid  the  USER  ASSISTANT.  It  supports  two 
types  of  interaction.  One,  available  only  to  direct  use  by  the  user,  provides 
a  documentation  "tree"  through  which  the  user  can  browse  in  a  more  or  less 
top-down  manner.  In  this  mode,  the  documentation  provides  a  wel 1 -organized 
on-line  reference  manual.  The  second  type  of  interaction  is  query-based  ard 
is  available  both  r.o  the  user  and  the  other  modules.  Inputs  in  a  query 
language  are  used  to  specify  database  search.es  and  the  result,  is  given  to  the 
invoking  source. 

Roth  the  query  language  and  the  result,  are  structured  in  a  machine- 
readable  form  to  allow  uniform  use  by  the  other  modules.  Those  modules 
determine  how  to  display  or  otherwise  use  the  results  in  their  own  interact!' 
context.  !  hi  Hi  I.  P  FACILITY  has  its  own  user  interface  for  allowing  the  user 
to  compose  queries  in  a  more  natural  mode  of  expression  or  for  displaying 
results  in  a  consistent  human-readable  form,  the  input  interface  includes 
both  closed  (menu)  and  open  (command  language)  input  mode  alternatives  for 
flexibi  1  i  ty. 

CAT  :’AC ! '  ! T Y .  The  CAI  FACILITY  uses  an  a_d  hoc  frame-oriented  approach  (the 
CAI  analog  to  a  programmed  text)  for  delivering  a  tutorial  introduction  an  the 
use  of  the  IKAS.  Much  of  the  content  of  instruction  is  retrieved  from  the 
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documentation  database  in  the  HELP  FACILITY.  Additional  content,  such  as 
examples  and  questions  to  test  understanding,  are  stored  as  a  separate 
curriculum  data  base  within  the  CAI  FACILITY. 

In  our  design,  the  CAI  FACILITY  is  not  integrated  with  the  main  IKAS 
modules.  Thus,  it  can  not  dynamically  invoke  '■hose  modules  to  present  "live1' 
examples,  nor  can  it  obtain  any  information  about  the  errors  the  user  makes  in 
initially  using  the  system,  when  he  may  switch  between  "playing"  with  the 
system  and  accessing  the  CAI  FACILITY.  It  is  limited  therefore  to  simulated 
examples  and  in  its  responsiveness  to  the  user's  particular  situation. 

We  do  not  believe  that  a  more  sophisticated  CAI  FACILITY  is  necessary  for 
those  IKAS  concepts  that  entail  significant  interaction  between  the  DE  and  KE 
prior  to  DE  use  of  the  IKAS  (i.e..  Alternatives  1  and  2).  Under  Alternative 
3,  a  more  integrated,  powerful  CAI  module  may  be  required  to  provide  self- 
contained  initial  access  by  users.  Work  on  an  initial  IKAS  development  effort 
can,  however,  proceed  independently  of  requirements  for  CAI  capabilities. 

SYSTEM  FEATURES 

The  system  architecture  described  above  is  intended  to  satisfy  several 
design  constraints.  These  include: 

MIXED  INITIATIVE.  Knowledge  acquisition  is  either  system-di rected  via  the 
EL  I C I T OR  or  user-di rected  via  the  EDITOR.  The  user  can  take  control  of 
initiative  at  any  time  or  pass  the  initiative  to  the  EL  I C I  TOR . 

DYNAMIC  CONTROL  OF  INITIATIVE.  The  flexible  mixed-initiative  interaction  is 
made  possible  by  dynamic  planning  of  the  system's  knowledge  acquisition 
objectives.  The  PLANNER  uses  context  information  saved  by  the  RECORDER  to 
maintain  an  agenda  of  knowledge  acquisition  topics.  Knowledge  acquisition 
objectives  can  thereby  be  altered  as  necessary  to  reflect  the  outcome  of  prior 
interactions,  whether  they  were  system-  or  user-controlled. 

CONCEPTUAL  SUPPORT  FOR  THE  USER.  The  ELICITOR  uses  cl  ass-generic  knowledge 
from  the  knowledge  base  and  domain-specific  knowledge  already  entered  by  the 
user  to  generate  and  interpret  user  inputs  using  abstractions  consistent  with 
the  user's  description  of  the  domain.  The  EXERCISER  enables  the  user  to 
inspect  and  invoke  the  host  performance  system — the  KB  IS— to  test  the 
performance  of  the  system  using  the  current  knowledge  base.  In  addition,  the 
iiri.P  and  CAI  facilities  can  include  reference  and  tutorial  information  for 
supporting  the  user’s  understanding  of  the  system. 

MODULARITY.  Functions  for  changing  and  using  the  knowledge  base  share  common 
resources  for  supporting  user  access,  for  checking  inputs  for  possible  errors, 
for  recording  context,  and  for  actual  access  to  the  permanent  knowledge  base. 
Different  aspects  of  these  functions  are  accomplished  by  more  than  one 
resource  (e.g.,  context  management  as  a  function  of  time  by  the  RECORDER  and 
the  MANAGER).  Modularity  is  of  course  desirable  in  almost  all  system 
archi tectures .  One  advantage  of  modularity  in  this  application  is  that  it 
should  enable  incremental  system  development,  thereby  enhancing  feasibility. 
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It  also  provides  both  greater  consistency  and  flexibility  in  the  user 
interface. 

CONSISTENCY.  The  MANAGER  mediates  all  interactions  between  the  various 
modules  and  the  knowledge  base.  Both  the  MANAGER  and  RECORDER  free  the  user 
from  concerns  about  how  to  access  and  update  the  knowledge  base  by  providing  a 
uniform  access  mechanism  to  all  the  modules.  Differences  in  implementation  of 
knowledge  base  access  from  these  modules  should  thus  be  invisible  to  the  user. 
For  error  checking  and  context  recording,  the  CHECKER  and  RECORDER  should 
apply  uniform  mechanisms  regardless  of  whether  they  are  invoked  by  the 
EL  IC I  TOR  (system  initiative)  or  the  EDITOR  (user  initiative).  The  EL I C I  TOR 
where  possible  allows  user  Inputs  to  be  accepted  via  the  EDITOR,  providing  the 
user  with  a  consistent  input  interface  under  both  system  and  user  initiative. 
Finally,  the  USER  ASSISTANT  provides  consistent,  low-level  monitoring  and 
intervention  for  all  user  inputs  to  the  major  user-accessible  functions. 

FLEXIBILITY.  Flexibility  is  inherent  in  the  user's  ability  to  shift  the 
responsibility  for  initiati  <e  at  any  time  to  use  either  a  special  ELICITOR 
input  protocol  or  the  EDITOR  when  the  ELICITOR  is  in  control.  Additional 
flexibility  can  be  achieved  through  the  USER  ASSISTANT,  which  can  implement 
low-level  lexical  and  syntactic  error  correction  and  support  a  consistent  set 
of  alternative  I/O  protocols  (e.g.,  using  multiple  media,  partial  input 
sped fication,  display  formats)  for  the  user  to  select  when  interacting  with 
the  three  major  user  functions. 

TURN-KEY  ACCESSIBILITY  AND  USE.  Suitable  implementation  of  the  USER 
ASSISTANT,  the  HELP  FACILITY,  and  the  CAI  FACILITY  could  make  the  system’s  use 
independent  of  external  documentation  and  human  support.  The  major  burden  is 
on  the  CAI  facility  to  help  the  user  understand  the  performance  system's 
representation  of  the  domain  if  that  understandi ng  cannot  be  developed  via 
prior  interaction  with  a  KE. 
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SECTION  VII 
CONCLUSIONS 


During  the  course  of  this  project,  considerable  effort  addressed  a 
careful  analysis  of  the  role  of  the  knowledge  engineer  (K£)  in  the  process  of 
building  an  expert  system.  This  analysis  was  directed  toward  the 
determinat ion  of  feasible  concepts  for  automating  functions  now  performed  by 
human  KEs.  The  nature  of  the  knowledge  engineering  process  and  the 
practi t ioners 1  current  understanding  of  it  appear  to  limit,  at  least  for  the 
foreseeable  future,  the  scope  and  generality  of  possible  automation  of  the 
KE 1 s  role  in  knowledge  acquisition. 

The  KF  uses  knowledge  and  information  that  is  incomplete,  inconsistent, 
and  heuristic  in  attacking  his  objectives.  part  of  the  problem  lies  in  the 
coronuni cation  gap  that  exists  initially  between  the  KE,  the  customer,  and  the 
DE.  Another  part  is  due  to  the  fact  the  knowledge-based  systems  technology— 
although  it  has  been  applied  to  some  real-world  problems— is  still  immature, 
lacking  the  breadth  and  depth  of  applicability  needed  for  the  emergence  of 
systematic,  general  methods.  As  a  result,  the  knowledge  engineering  process 
is  iterative  and  incremental  ,  with  experience  gained  early  in  the  process  used 
in  subsequent  stages  to  refine  and  revise  system  objectives  arid  behavior. 

The  iterative,  incremental  nature  of  knowledge  engineering  implies  that 
knowledge  acquisition,  a  single  objective  in  the  entire  process,  cannot  be 
isolated  from  other  objectives  by  automation  without  interfering  with  the  KI  's 
pursuit  of  those  other  objectives.  Thus,  stand-alone  or  near  stand-alone 
automation  for  knowledge  acquisition  in  building  a  knowledge-based  system  is 
not  feasible  at  the  present. 

Given  this  conclusion,  we  developed  a  system  concept  for  automating 
knowledge  acquisition  that  avoids  a  gene, al ,  comprehensive  approach  in  favor 
of  a  more  feasible,  usable  alternative.  The  concept  is  oased  on  the  notion  of 
a  class  of  related  tasks.  It  proposes  that  once  a  knowledge-based  syst.v  s 
implemented  "manually"  for  one  task  in  the  class — thereby  achieving  all  cm. 

KE 1 s  objei  ! i ves -- i t  would  be  possible  to  automate  elicitation  from  domain 
experts  of  knowledge  liases  for  other  tasks  ir  the  class.  These  knowledge 
bases  would  be  used  ,n  a  system  with  t.ho  same  capabilities  and  arch)  tec  turn  as 
t.he  first,  system.  The  knowledge  acquisition  mechanisms  would  mala1  use  of 
knowledge  and  information  obtained  in  the  first  effort;  hence,  they  would  be 
specific  to  that  knowledge-based  systems  architecture  for  that  class  of  tasks. 
Automated  know  lee-go  acquisition  would  still  proceed  incrementally  arid 
iteratively  throughout  stages  of  formal ization ,  implementation,  and  testing. 
However,  only  the  knowledge  base  of  the  system  would  be  affected  by  this 
process:  all  other  aspects  of  system  design  arid  function  would  remain  fixed. 

Although  limited  in  scope  and  generality,  r'rs  approach  to  automating 
knowledge  acquisition  should  be  worthwhile  for  isses  with  many  members  or' 
for  systems  where  new  knowledge  must  be  added  guently  over  the  life  of  tt 
system.  It  builds  upon  prior  research  on  assistance  for  knowledge 
engineering,  which  adds  to  its  credibility.  However,  it  involves  sol  vino 
significant  new  problems.  Perhaps  the  most  important,  of  these  are  (I)  how  to 
identify,  formalize,  and  use  class-generic  abstractions,  and  (?)  how  to 
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provide  conceptual  support  to  users  that  would  enable  OFs  to  use  the  system 
d  i  •*ec  1 1  y . 

The  tore11  .il  terrwtive  appli  rations  of  the  automated  knowledge  acquisition 
concept  presented  in  Section  IV  have  different  costs  arid  expected  benefits 
according  to  the  criteria  considered.  However,  each  appears  to  present  a 
promising  and  productive  avenue  for  realizing  new  capabilities  in  automated 
knowledge  acquisition,  Mon- technical  considerations  regarding  organizational 
needs  or  synergy  with  on-going  research  programs  nay  ultimately  influence 
which  al tern, .live  would  be  mosf  fruitful  to  pursue.  Although  our  concept 
definitions  were  shaped  primarily  by  technical  considerations,  we  also  were 
influenced  by  our  perceptions  of  those  contributions  that  would  provide  high 
value  to  the  Navy  at  this  time. 

We  also  determined  that  a  detailed  design  of  the  user  interface  for  an 
I K AS  is  not  possible  without  commitment  to  a  specific,  detailed  KBIS-IKAS 
archi tecture .  However,  our  discussions  with  Navy  system  builders  and  DEs 
indicated  that,  while  the  details  of  an  IKAS'  user  interface  are  not 
inconsequential,  they  will  probably  not  determine  whether  Navy  DEs  can 
effectively  use  initial  IKAS  technology.  Instead,  selection  of  users  and 
high-level  user  interface  character!' sties  not  identified  with  particular  media 
or  interaction  protocols  are  more  critical  for  a  successful  IKAS 
implementation  for  a  Navy  training  system.  We  therefore  made  the  following 
recommendations,  reflected  in  the  architecture  proposed  in  Section  VI,  for 
pursuing  further  design  of  an  IKAS- 

a.  Emphasize  conceptual  support  and  transparency . 

b.  Design  the  knowledge  acquisition  strategy  to  tie  modular  with 
respect  to  acquiring  different  types  of  knowledge. 

:.  Pro  aide  extensive  on-line  hel p/doc umentati on  facilities  and 
intelligent  context  management . 

d.  Defer  comnitments  on  other  user  interface  issues  in  an 
incremental  design  and  development  ipproach. 

We  believe  that  by  following  these  recommendations  further  research  can 
build  upon  the  results  of  the  present  projeef  to  develop  a  first  effective 
erotutype  IK  id. 
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APPENDIX  A 

ILLUSTRATIVE  INSTRUCTIONAL  KNOWLEDGE  ACQUISITION  DIALOGUE 


This  Appendix  presents  a  hypothetical  dialogue  between  a  DE  and  an  KE  who 
are  working  together  to  define  knowledge  to  be  used  by  a  KBIS  for  surface  Navy 
tactics  training.  It  is  intended  to  illustrate  some  of  the  requirements  for 
eliciting  knowledge  used  for  trainee  performance  modeling  (see  IKAS 
Alternative  1  in  Section  IV).  The  dialogue  assumes  the  KE  and  DE  have  been 
interacting  for  a  considerable  time  and  that  a  competence  model  is  largely 
completed. 


★  ★  ★  ★  ★ 


KE:  We've  considered  the  procedures  and  rules  for  successful  defense  oF  a 
sTngle  ship  from  surface  and  air  threats.  Now  let's  consider  how  things  can 
be  done  wrong  if  a  TAO  lacks  that  knowledge.  In  particular,  I  want  you  to 
think  about  training  exercises  you've  supervised  and  the  errors  you've 
frequently  seen  and  what  causes  them.  Let's  limit  ourselves  first  to  "weapons 
free"  situations.  You  Indicated  earlier  that  decisions  were  contingent  on  how 
strongly  you  believed  your  opponent  had  sufficient  data  to  target  you.  Are 
there  Important  errors  that  you've  seen  there? 

DE:  Well,  yes.  It's  mostly  a  case  of  keeping  track  of  his  and  your  emissions. 
T”guess  one  common  mistake  Is  to  forget  about  emissions  you  may  have  produced 
before  you  knew  he  was  there,  like  HF  or  even  UHF  conmuni cations.  I  think  I 
told  you  that  If  you  detect  him  passively  and  you  have  put  out  some  emissions 
you  should  start  a  zig-zag  If  you  are  authorized  to. 

KE:  So  in  that  type  of  situation,  not  seeing  a  zig-zig  would  lead  you  to 
Relieve,  as  an  observer,  that  a  TAO  either  had  forgotten  about  the  emissions 
or  didn't  know  that  a  zig-zag  was  the  thing  to  do  then? 

DE:  Yes,  most  of  the  time.  Depending  on  what  type  of  opponent  he  believes  he 
Ts  facing,  he  could  decide  it  was  very  unlikely  that  the  opponent  could  get 
targeting  Information  passively  from  the  emissions.  In  that  case,  he  might 
not  zig-zag.  And  remember,  you  can't  always  zig-zag;  it  depends  on  the 
formation  and  on  whether  you  happen  to  be  on  a  ship  with  certain  types  of 
towed  sonar  deployed. 

KE:  Right,  I  can  see  that  here  In  ny  notes  about  manueverlng.  You  gave  me 
Those  constraints  before. 

DE:  Remember  too  the  deception  angle.  If  you  really  think  he  has  targeting 
lata  and  is  just  waiting  to  get  a  better  shot,  then  you  might  buy  some  time  tc 
get  your  own  systems  ready  to  fire  by  NOT  zig-zagging.  So,  It's  not  all  that 
simple  to  say  from  not  seeing  zig-zagging  that  a  TAO  has  not  been  thinking 
about  his  emissions.  It's  a  matter  of  judgment.  Like  I  told  you,  you  don't 
want  to  be  too  cute.  If  the  situation  is  hot  and  you  gave  him  some  passive 
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data  and  now  you  have  some  from  him,  you  really  want  to  think  about  going 
active  first  to  get  targeting  data. 

KE :  So,  the  way  you  would  really  evaluate  a  TAO's  performance  is  whether  he 
Tet  the  opponent  get  off  the  first  shot  on  a  good  intercept  heading? 

DE:  Yes,  it's  pretty  imprecise— sometimes  you  can  weigh  the  odds  as  best  you 
can  and  take  the  actions  you  know  are  best  and  still  come  out  on  the  short 
end.  I'd  need  to  see  a  guy  apparently  miss  the  fact  that  he  had  been  targeted 
several  times  before  I'd  conclude  that  he  was  not  considering  his  emissions 
history  or  acting  on  it  to  the  best  of  his  ability.  Of  course,  if  it  came  up 
in  a  training  exercise,  I  could  ask  him  about  it  in  the  debriefing  if  it 
seemed  to  impact  the  outcome. 

KE:  Suppose  you  wanted  to  figure  out  whether  a  TAO  trainee  using  a  training 
sTmulator  like  NAVTAG  had  this  problem  just  by  looking  at  his  performance. 

What  type  of  situations  would  you  set  up? 

DE:  I'd  put  him  on  a  low-capable  ship  and  give  him  some  INTEL  about  some  high- 
capable  opponents;  that  way  he  wouldn't  be  quick  to  go  on  the  offensive 
without  first  hoping  ORANGE  would  give  him  some  good  data  that  could 
compensate  for  the  capability  imbalance.  I’d  put  no  constraints  on  his 
manuevering.  Then  I'd  set  him  up  by  having  him  respond  to  some  conmuni cation 
and  at  some  later  time  pick  up  some  ambiguous  passive  emission  that  was 
ORANGE-originated  but  insufficient  for  identification.  Then  I  could  look  for 
a  zig-zag  with  more  certainty.  I'd  give  him  a  few  exercises  in  which  those 
were  the  features  before  any  engagement  actually  commenced. 

KE:  What  about  his  possible  belief  that  the  enemy  hadn't  been  able  to  target 
TiTm,  or  an  attempt  at  deception? 

DE:  Well,  on  the  first,  those  high-capability  ships  generally  have  the  best 
E5M  systems,  so  if  his  INTEL  says  that's  what  he  might  expect,  he  wouldn't 
want  to  ignore  them.  Also,  instead  of  just  a  COMM  emission  I  could  set  the 
simulator  to  give  him  a  real  emissions  error— tell  him  his  ECM  equipment  or  a 
weapons  control  radar  accidently  went  on  for  a  few  seconds.  That's  real 
unlikely,  but  possible.  As  to  the  deception,  if  I  set  up  the  trainer  to 
ignore  his  deception  and  hit  him  with  an  SSM  if  he  didn't  zig-zag,  then  after 
a  few  exercises  he'd  be  zig-zagging  if  he  knew  he  was  supposed  to  do  so  or  if 
he  wasn't  forgetting  his  emissions  history.  Only  problem  with  that  is 
discouraging  his  use  of  deception  when  it  might  be  his  best  chance.  I'd 
address  that  directly  if  I  were  an  instructor  after  I  was  sure  a  trainee  knew 
the  best  non-deception  response  to  the  situations. 

KE:  Let's  talk  about  situations  now  where  BLUE  believes  he's  been  targeted 
wTth  regard  to  bearing  and  wants  to  Initiate  the  engagement.  Those  are  where 
he  is  on  a  high-capable  platform  or  has  an  important  defensive  role  for  some 
other  HVUs. 

DE:  That's  right.  The  biggest  problems  are  not  giving  ORANGE  better  targeting 
{lata  before  you  are  ready  to  fire  yourself. 

KE:  Let's  talk  first  about  the  case  where  BLUE  hasn't  seen  a  surveillance 
radar  emission  from  ORANGE. 


.  ..  . 
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DE:  Well  one  problem  I've  seen  there  Is  BLUE's  use  of  his  own  surveillance 
radar.  First,  you  don't  want  to  use  it  if  your  EW  people  tell  you  the  track's 
source  seemed  to  be  at  a  range  outside  your  destruction  zone.  If  you  do,  all 
you  are  going  to  do  is  confirm  your  identity  and  maybe  give  him  track  info. 

It  may  allow  you  to  establish  a  better  track  on  him  but  you  can't  do  anything 
about  it. 

KE:  You  told  me  that  in  that  case  you  should  steam  toward  the  target's 
Bearing,  so  that  the  error  is  in  not  executing  that  procedure  when  the 
situation  warrants  it— that  is,  when  you  have  suggestive  evidence  that  the 
target  is  out  of  range. 

DE:  Right.  Except  of  course  in  the  case  where  your  SSMs  are  mounted  aft  and 
you  might  not  be  able  to  recover  fast  enough  to  shoot  if  you  are  in  range  and 
he  shoots  first.  That's  the  tricky  one  if  you  are  determined  to  be  offensive. 

KE:  You  said  “first"  problem  before:  what  other  problems  are  there  in  using 
surveillance  radar  at  that  point? 

OE:  It's  turning  it  on  continuously  right  away  instead  of  taking  a  snapshot. 
That  gives  him  better  targeting  and  all  you  need  is  a  snapshot  to  determine 
how  to  engage  him  with  the  fire  control  systems.  You  don't  want  to  monitor 
with  your  surveillance  radar  unless  you've  already  seen  his  fire  control  radar 
light  off. 
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